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ABSTRACT 

Technological  change  and  the 
renewed  focus  on  military  capability 
(rather  than  equipment)  as  the  key 
driver  of  acquisition  are  creating  a 
more  “joined-up”  environment  for  T&E. 
The  talk  reflects  on  the  implications  of 
this.  Examples  of  computer-based 
visualisation  of  trials  results  are  used 
to  illustrate  how  technology  is 
contributing  to  improved  practice  in 
T&E  and  the  delivery  of  enhanced 
value  to  customers. 

INTRODUCTION 

Although  I  am  a  relative  newcomer  to 
the  T&E  business,  1  was  struck 
yesterday  by  the  focus  that  this 
community  puts  on  the  requirements 
of  the  war-fighter.  Recognising  that 
T&E  is  a  means  to  an  end,  and  not  an 
end  in  itself,  puts  the  community  in 
good  position  to  adapt  and  respond  to 
the  rapidly  changing  environment,  the 
theme  of  today’s  sessions. 

The  programme  for  this  event  has 
been  deliberately  structured  so  as  to 
enable  us  to  debate  the  issues  in 
context  and  this  leads  me  to 
acknowledge  the  hard  work  of  the 
many  organisers  and  presenters  who 
have  brought  the  event  into  being.  I 
know  very  well  the  considerable  effort 
it  takes  to  prepare  papers  and  support 
the  organising  committees  and  I  pay 
tribute  to  all  who  have  clearly  done 
such  a  fine  job. 

My  brief  this  morning  is  then  to  focus 
us  on  the  changing  environment  and 
the  challenges  -  or  rather  the 
opportunities  -  which  this  presents  in 


terms  of  roles,  practices  and 
technologies.  I  aim  to  do  this  by 
looking  at  2  of  the  dominant  drivers  for 
T&E:  the  focus  on  military  capability 
and  prospects  of  technological 
improvement.  Let  me  briefly  lift  the  lid 
on  each  of  these. 

MILITARY  CAPABILITY 

Firstly,  I  want  to  make  some 
observations  on  the  way  in  which  a 
capability-centred  view  amplifies 
trends  towards  complexity  and 
concurrency  that  were  already  present 
in  defence  acquisition.  These  are  the 
result  of,  for  example,  a  generally  less 
intense  but  far  more  unpredictable  set 
of  threats,  rapid  succession  of 
generations  of  electronic  and  software 
systems,  COTS  components  and  their 
associated  obsolescence  issues, 
extended  platform  life  and  of  course 
digitisation  of  the  battlespace,  with  its 
effect  of  joining  almost  everything  to 
everything  else.  At  the  single 
equipment  level,  one  may  well  be 

dealing  with  multiple  concurrent 
upgrade  cycles  as  well  as 

dependencies  on  other  equipments 
without  which  the  first  equipment 
cannot  deliver  worthwhile 
performance.  Those  other  equipments 
might  well  be  C4/ISTAR  assets 
comprising  a  mix  of  legacy  and  new 
equipments,  all  subject  to  some  form 
of  incremental  acquisition.  The 
capability  viewpoint  extends  this  to 
different  classes  of  equipment, 

possibly  operating  in  different 

environments  in  classical  terms  but  all 
contributing  to  the  military  capability 
concerned  and  probably  to  others  too. 
The  point  of  reminding  you  of  all  this  is 
that  it  describes  a  highly  complex  and 
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concurrent  environment  in  which 
decisions  and  judgements  cannot 
sensibly  be  made  in  isolation. 
Research,  Development,  Technical 
Assistance,  OA  and  T&E  will  have  to 
be  significantly  “joined  up”  than 
hitherto,  which  could  cause  some 
discomfort  to  established  attitudes. 
Does  this  give  the  traditional 
adjudicators  significant  ownership  and 
responsibility  over  the  result  of  the 
acquisition?  Does  it  matter? 

The  shift  towards  an  holistic  vision  of 
military  capability  gives  us  the 
opportunity  to  join-up  our  T&E  forces 
with  those  of  other  acquisition  players 
to  support  a  single  goal  of  military 
effectiveness.  There  is  nothing  new  of 
course  in  ‘jointery’;  the  US  has  for  at 
least  two  decades  been  successful  in 
combining  DT&E  and  OT&E  and  the 
UK  has  done  quite  well  in  conducting 
‘purple’  T&E  on  air  systems.  However, 
it  is  perhaps  only  when  we  all  focus  on 
a  single  high  level  goal  that  we  truly 
realise  the  complexity  of  the 
undertaking.  Already,  with  Smart 


Procurement,  we  have  seen  the 
adoption  of  the  systems  engineering 
(SE)  approach  as  a  pragmatic 
response  to  managing  acquisition 
complexity.  It  does  this  by  thoroughly 
tracking  the  interdependencies  and 
taking  proper  account  of  the  impact  on 
military  capability  when  making  design 
or  programme  decisions.  Of  course, 
there  is  nothing  fundamentally  new 
about  this  as  a  concept,  but  the 
availability  of  effective  computerised 
toolsets  to  support  the  process  makes 
it  a  good  deal  more  practical  to  apply 
the  necessary  rigour  in  practice.  We 
should  look  to  apply  these  principles  to 
the  management  of  T&E.  In  Figure  1, 
I  have  represented  four  planes  on 
which  T&E  will  always  be  at  work; 
these  are  concept  demonstration, 
development  &  qualification; 
equipment  effectiveness  and  military 
capability. 


T&E  directed  at  Military  Capability 


Scientists 


I ntegrated  Thrqugh-life  T^E  Plqn 


War 

Fighters 


Fig  1 
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The  natural  tendency  in  prosecuting  the 
T&E  task  is  to  think  from  the  left-front- 
bottom  corner  and  expand  the  T&E 
activity  in-line  with  developing 
requirements.  Systems  Engineering 
informs  us  though  that  we  need  to  be 
much  more  rigorous  in  capturing  and 
engineering  our  requirements  against  the 
high  level  product  -  military  capability.  By 
examining  these  early,  across  all  four 
planes  we  will  start  to  generate  a 
comprehensive  set  of  requirements  that 
can  be  addressed  by  means  of  a  fully 
integrated  T&E  plan.  As  well  as  optimising 
the  T&E  solution  this  approach  also  has 
significant  potential  to  influence  system 
design.  For  example,  it  is  almost  certainly 
more  cost  effective  to  design  for  test  at  the 
outset  rather  than  incur  the  cost,  time  and 
flexibility  penalties  associated  with 
special-to-test  instrumentation  fits.  If,  at 
the  same  time,  we  address  measurement 
requirements  of  training  (and  indeed 
engineering  support)  we  start  to  see  the 
potential  of  this  approach.  We  cannot  be 
involved  in  such  potentially  worthwhile 
judgements  without  a  well-informed 
understanding  of  the  requirements  at  all 
levels.  This  is  the  more  “joined  up” 
approach  that  I  referred  to  earlier. 
However,  it  is  not  for  the  faint  hearted, 
because  it  implicates  the  T&E  engineer  in 
the  ultimate  achievement  of  the  desired 
military  capability  in  ways  that  traditional 
methods  of  working  did  not. 

The  process  for  managing  integrated  T&E 
is  highly  amenable  to  the  systematic 
approach  inherent  in  systems  engineering. 
By  continuously  reviewing  and  updating 
goals  at  all  levels,  scarce  T&E  resources 
can  be  directed  at  priority  areas  of 
technical,  programme  and  cost  risk.  This 
stands  in  contrast  with  the  procedurally 
based  approach  which  has  everyone 
fulfilling  traditional  roles  in  isolation  without 
properly  weighting  the  priorities  or 
impacts.  Integrated  T&E  brings  other 
challenges  too.  Almost  by  definition, 
integration  demands  trust  and  respect 
between  the  players.  That  is: 

•  understanding  and  accepting 
each  others  roles,  even  when 


these  are  necessarily  in 
tension: 

•  sharing  all  relevant  information 
and  data; 

•  protecting  other  parties’  IPR; 

•  agreeing  common  goals 
against  end  user  requirements 

There  are  some  difficult  cultural  issues 
and  I  will  be  interested  to  here  other  views 
on  these  during  the  plenary  session. 


TECHNOLOGICAL  IMPROVEMENT 

Having  talked  in  broad  terms  about  the 
focus  on  military  capability,  I  plan  to  use  a 
specific  example  to  illustrate  the  way  that 
technological  improvement  is  being 
embraced  within  the  UK  T&E  community. 

I  will  do  this  by  considering  the  recent 
application  to  T&E  of  the  technology  of 
visualisation. 

Visualisation  is  a  pseudo-3D  graphical 
representation  by  computer,  of  information 
derived  from  the  real  world.  The 
technique  can  be  applied  in  several  ways 
to  the  testing  of  complex  weapons 
systems  and  platforms  and  to  mission 
rehearsal  and  training,  providing  a  rapid 
and  cost  effective  means  of  transferring 
knowledge. 

What,  then,  are  the  benefits  of  using  this 
technology  in  a  T&E  scenario?  In  the  first 
instance,  it  can  ease  the  task  of  trials 
planning  and  preparation.  It  allows 
problem  solving  before  the  event  itself 
takes  place,  by  viewing  the  intended 
location  and  assets  from  multiple 
perspectives.  It  can  be  used  as  a  pre-trial 
training  tool  for  participants,  thus  reducing 
the  risk  in  the  programme. 

If  the  images  on  the  screen  are  driven  by 
the  data  from  tracking  instrumentation, 
visualisation  can  also  allow  real-time  trial 
observation  of  the  sequence  of  events. 
This  is  particularly  useful  for  underwater  or 
air-to-air  engagements,  which  may 
otherwise  be  unobservable. 
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Now  that  the  use  of  T&E  assets  is  being 
extended  to  mission  rehearsal, 
visualisation  can  play  a  significant  part  in 
familiarisation  of  aircrew,  for  example,  with 
the  practice  mission  before  it  is  flown. 
This  applies  equally  to  the  normal  conduct 
of  trials. 

In  cases  where  trial  data  sets  are  used  to 
build  and  validate  models,  visualisation 
assists  with  step-through  post  trial 
analysis,  enabling  normally  inaccessible 
properties  to  be  viewed  by  fusion  of  trial 
data  with  visual  images. 

Training  now  forms  much  of  the  business 
of  the  UK  T&E  Sectors,  and  visualisation 
has  been  applied  successfully  to  post 
sortie  analysis,  speeding  the  development 
of  operational  tactics  by  means  of  rapid 
after-action  review. 

Finally,  visualisation  can  be  driven  by 
telemetry  data  to  allow  more  precise 
control  of  the  systems  involved  in  a  trial. 

Let  us  take  a  look  at  some  examples  of 
these  developments... 

Trials  planning  and  risk  reduction 

This  visualisation  of  a  bombing  range  in 
Norway  produced  for  the  RAF  is  being 
used  to  assist  in  trials  planning  and 
preparation.  As  it  is  possible  to  view  the 
terrain  from  any  position,  one  is 
immediately  able  to  assess  what  problems 
may  develop  from  the  placement  of 
sighting  equipment  (Electro-optic  trackers, 
etc.)  during  a  proposed  trial  in  this 
extreme  terrain.  Furthermore,  pilots  can 
be  introduced  to  the  layout  of  the  range 
and  be  shown  the  routes  to  be  flown  in  a 
trial,  gaining  a  good  understanding  of  what 
to  expect  and  do,  before  entering  a  foreign 
facility. 

The  total  visualisation  details 
approximately  400  sq.  km  of  mountainous 
terrain  using  DTED  level  1  data  producing 
9200  polygons  over  which  a  map  showing 
roads,  buildings,  contour  lines  and  other 
geographical  points  of  interest  is  placed. 


Note  the  F18  used  in  this  visualisation  is 
enlarged  to  five  times  its  normal  size  to 
aid  the  presentation. 

Real-time  engagement  observation 

The  visualisation  of  the  British  Underwater 
Test  and  Evaluation  Centre  is  currently 
being  completed.  It  was  designed 
originally  for  a  quick  debrief  on  trial 
events,  but  is  being  adapted  to  enable 
tracking  system  data  to  be  fed  into  the 
visualisation  in  real  time,  allowing  remote 
observation  of  engagements  in  progress. 

This  visualisation  is  based  on  an  accurate 
database,  generated  from  underwater 
soundings  of  the  seabed,  and  from  DTED 
Level-1  data  of  the  surrounding  land.  This 
detail  allows  strategic  operational  centres, 
such  as  the  range  control  buildings, 
hydrophones,  and  other  monitoring 
equipment  to  be  added  quickly  to  the 
database. 

It  is  now  possible  to  view  the  submarine, 
torpedo,  target  and  all  events  as  they 
happen,  from  any  perspective,  thus 
providing  a  major  aid  to  understanding  of 
the  results. 

Mission  rehearsal 

With  growing  pressure  for  our  Armed 
Forces  to  perform  operations  with  minimal 
collateral  damage,  mission  rehearsal  is 
becoming  of  major  importance.  By 
creating  a  detailed  and  accurate  database 
of  one  of  our  airfields  and  the  surrounding 
range  area,  its  use  for  mission  rehearsal 
can  be  assisted.  With  precise  scaling  and 
placement  information,  it  is  possible  to 
view  events  in  real  time  or  quickly  after 
they  have  been  recorded.  This  can  be 
used  to  prepare  aircrew  for  a  mission 
being  simulated  on  the  range,  or  for  a 
quick  after-action  review. 

The  model  can  easily  be  edited  using  the 
appropriate  software,  and  proposed  trials 
or  actual  data  quickly  incorporated.  The 
scaling  is  such  that  one  unit  in  the  mode! 
represents  one  metre,  and  the  placements 
of  the  bombing  pads,  runways,  and 
buildings,  correspond  to  their  positions  on 
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Ordnance  Survey  maps.  The  images  used 
in  the  model  were  extracted  from  digital 
photographs  or  from  digital  video. 

Model  building  and  validation 

Here  we  have  a  visualisation  of  a  test 
firing  of  a  155mm  shell  from  an  AS90  gun, 
over  a  Multiple  Receiver  Doppler  Radar 
system  (MRDR).  MRDR  can  be  used  to 
determine  the  movement  of  the  shell  in  all 
three  planes  and  in  roll,  pitch  and  yaw, 
without  the  need  for  mathematical 
integration.  This  motion  can  accurately 
displayed  within  a  visualisation,  which 
assists  in  validating  a  six-degree-of- 
freedom  model.  As  any  imaginable 
viewing  position  can  be  obtained, 
properties  can  be  viewed  which  are 
normally  inaccessible  due  to  engineering 
limitations  or  safety  considerations.  We 
can,  in  effect,  fly  with  the  shell,  studying  its 
motion  from  behind  and  alongside. 

Post-trial  or  post-training  analysis 

This  application  of  the  visualisation 
process  gives  an  example  of  how  it  can 
assist  in  the  comprehension  of  complex 
high-speed  interactions.  Not  only  is  the 
information  on  the  flight  of  the  aircraft, 
target  and  missile  in  this  theoretical 
engagement  brought  to  the  viewer  via  a 
graphical  representation,  but  also  an 
informative  2D  overlay  is  included  to 
provide  details  of  various  important 
variables.  For  example  the  system  can  be 
used  to  analyse  an  air-to-air  weapon 
firing,  the  progress  of  which  can  be 
reviewed  on  screen.  Development  and 
demonstration  of  amendments  to  the 
tactics  can  be  greatly  assisted. 

One  of  the  most  profitable  applications  of 
this  technique  is  post-sortie  analysis 
during  missile  practice  camps.  It  is  even 
feasible  that  the  scenario  could  be 
changed  while  the  aircraft  and  the  target 
were  still  airborne,  as  a  result  of  the 
observations  made  in  real  time. 

Real-time  control  of  systems 

In  this  final  example,  the  data  relayed  be 
telemetry  from  an  unmanned  aircraft  has 


been  visualised  to  assist  with  its  control. 
As  you  can  imagine,  controlling  a  platform 
of  this  type  from  a  remote  ground  station 
is  not  easy.  By  providing  the  operators 
with  a  moving  display  as  shown,  they  are 
much  more  easily  aware  of  the  attitude 
and  status  of  the  platform  ant  the 
enhanced  targets  which  it  tows. 

There  are  several  benefits,  which  include 
reduced  training  time  for  operators, 
reduced  risk  of  abortive  sorties  during 
trials  or  missile  practice  camps,  and 
improved  safety. 

I  hope  the  example  of  visualisation  has 
demonstrated  the  way  in  which  T&E 
techniques  are  being  adapted  to  reflect 
the  changing  nature  of  military  operations. 
As  many  colleagues  at  home  and  abroad 
have  observed,  we  need  to  address  the 
need  for  operational  T&E  seriously  if  our 
assets  are  to  continue  to  be  of  assistance. 
There  is,  in  addition,  ever  more  emphasis 
on  the  use  of  T&E  facilities  for  training. 
The  frequent  use  of  models  and 
simulations  to  aid  decisions  in  the 
equipment  acquisition  process,  demands 
a  T&E  capability  which  is  of  immediate 
application  in  developing  and  validating 
these  models.  We  expect  these  trends  to 
continue,  and  of  course,  we  shall  continue 
our  efforts  to  adapt  within  the  T&E 
community  as  history  unfolds. 
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Abstract. 

The  Pacific  region  is  home  to  some  of  the  largest 
air,  land  and  sea  ranges  in  the  world.  Their  missions 
include  ballistic  missile  and  intercept  testing,  air  and  sea 
launched  missile  testing,  drone  engagement  testing, 
conventional  weapons  and  munitions  testing,  electronic 
warfare  testing  and  joint  battle-space  environment  testing. 
In  the  current  Defence  climate  of  fewer  dollars,  fewer 
people  and  fewer  but  far  more  demanding  tasks,  the 
survival  of  their  capabilities  is  becoming  increasingly  at 
risk.  The  assertion  is  made  that  that  they  will  be 
incapable  of  meeting  the  demands  of  the  future  in  the 
absence  of  a  cooperative  investment  plan  that  is  prepared 
with  strategic  intent. 

This  paper  explores  the  role  that  the  International 
Test  and  Evaluation  Association  [ITEA]  has  played  in  the 
Pacific  region  to  improve  professional  dialogue  in  T&E, 
enable  relationships  to  be  developed  and  hence  catalyse 
the  taking  of  new  and  highly  successful  initiatives. 

I  am  hopeful  that  the  newly  formed  UK  Chapter 
will  be  encouraged  by  this  -  as  there  is  much  evidence  to 
suggest  that  most  of  us  are  trapped  in  the  ‘here  and  now’ 
with  little  spare  capacity  to  commit  the  resources  needed 
to  secure  the  capabilities  of  tomorrow.  This  obsession 
with  the  present  needs  to  change  —  before  some 
capabilities  are  lost  to  humankind  forever. 

Test  and  Evaluation  Activities  in  the  Pacific  Region 
Introduction 

A  claim  that  has  never  been  challenged  by  anyone 
I  know,  is  that  the  International  Test  and  Evaluation 
Association  (ITEA)  is  the  pre-eminent  professional 
society  for  staff  with  an  interest  in  Test  and  Evaluation 
(T&E)  -  whether  its  aerospace  (like  the  theme  of  this 
forum)  or  anything  else.  The  need  for  such  a  professional 
society  was  first  recognised  in  the  United  States  in  the 
early  1980’s,  where  errors  of  judgement  lead  to  the 
costly  failure  of  many  complex  military  systems  that  were 
being  developed  in  the  United  States  at  the  time.  And  as 
with  all  failures,  humankind  was  given  opportunity  to 
leam  from  those  mistakes  -  or  be  bound  forever  to  repeat 
them. 

ITEA  Successes 

With  the  establishment  of  ITEA,  opportunity 
therefore  presented  itself  to  leam  from  such  mistakes  - 
and  this  was  done  with  so  much  success  that,  in  1987, 
science  fiction  author  Isaac  Asimov  prophesied  that  the 
new  science  of  T&E  would  become  our  global  early 
warning  system  -  to  prevent  stargazers  from  tripping  over 
their  technological  feet. 


And  in  bringing  that  prophesy  into  reality,  I 
believe  that  the  United  States  learned  from  its  military 
mistakes  so  successfully  that  in  1987,  the  fundamental 
precepts  of  T&E  were  institutionalised  across  all  Federal 
Departments  via  an  edict  from  the  Office  of  Management 
and  Budget. 

Since  that  time,  this  focus  on  the  fundamental 
T&E  precepts  has  migrated  to:  the  Australasian  region 
through  the  establishment  of  the  Southern  Cross  Chapter; 
the  Middle  East  through  the  establishment  of  the  Israeli 
Chapter  and  to  Europe  through  the  vision  of  David 
Kimberley  and  the  establishment  of  the  UK  Chapter. 

A  Compelling  Need  for  Global  T&E  Dialogue 

I  strongly  believe  that  this  globalisation  of  ITEA 
and  the  global  appropriation  of  the  fundamental  T&E 
precepts  is  now  a  compelling  and  essential  response  to 
both  the  Revolution  of  Military  Affairs  and  the 
Revolution  of  Business  Affairs. 

For  none  of  us,  in  Government  or  Industry  are 
immune  from  this.  We  are  no  longer  dancing  to  national 
tunes  but  to  the  tune  of  rapid  globalisation  and  the  market 
pressures  and  competition  that  this  brings. 

As  a  compelling  example  of  commercial 
competition,  you  only  have  to  witness  the  radio¬ 
frequency  spectrum  auctions  where  spectram  critical  to 
both  national  and  coalition  force  operations  is  at  grave 
risk  of  being  lost  forever. 

To  the  present  day,  ITEA  members  have  been  the 
pioneers  at  the  fore-front  of  change  and  ITEA  itself  has 
been  the  catalyst  for  drawing  the  T&E  community 
together,  improving  the  availability  of  post-graduate 
education,  and  facilitating  professional  dialogue  via 
international  conferences  that  are  now  held  regularly  in 
three  of  the  major  world  regions;  the  Americas,  Europe 
and  the  Asia/Pacific. 

I  am  therefore  hopeful  that  this  presentation  will 
be  an  encouragement  to  the  UK  Chapter  -  because  in  the 
early  days  of  the  Southern  Cross  Chapter  formation  it  was 
like  firing  a  flare  from  a  life-boat  to  see  if  anyone  else 
was  out  there. 

Impact  of  Globalisation 

So,  other  than  encouraging  the  UK  Chapter,  what 
am  I  am  doing  as  the  ITEA  Pacific  region  representative 
at  a  European  T&E  Forum.  Well,  lessons  shared  are 
lessons  learned,  so  I’ll  be  sharing  some.  As  well,  I’ll  be 
taking  opportunity  to  tell  you  what  we’re  famous  for  in 
the  Pacific  -  so  we  can  be  considered  to  be  part  of  the 
global  solution  to  what  must  now  be  addressed  as  a  global 
problem  -  investment  planning  for  the  future. 

For  with  respect  to  the  future  of  T&E,  we  are  now 
talking  about  security  of  investment  in  national,  regional 
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and  global  resources  -  which  are  under  aggressive  attack 
from  commercial  interests  -  notably  airspace,  land-space, 
sea-space  and  of  particular  note  ‘radio-frequency 
spectrum  space’. 

As  an  example  of  the  latter,  recent  spectrum 
auctions  have  left  the  European  ranges  with  only  20MHz 
of  harmonised  telemetry  spectrum  with  which  they  are 
expected  to  test  their  weapons  systems  as  well  as  their 
space  and  aerospace  platforms  of  the  future  -  both 
military  and  civil.  20MHz  does  not  even  enable  an  inter¬ 
range  cooperative  drone  engagement  to  occur  today  -  let 
alone  consider  what  might  be  on  the  weapons  system 
drawing  boards  of  tomorrow.  And  believe  me,  this  is  only 
the  beginning.  So  please  support  the  newly  formed 
International  Consortium  on  Telemetry  Spectrum  [ICTS] 
to  help  defend  these  interests.'  For  the  future  of  space  and 
aerospace  endeavours  are  now  at  significant  risk  -  in  both 
military  and  civil  application.  And  it  doesn't  stop  there. 

All  of  the  essential  test  and  training  infrastmcture 
is  under  attack  so  I  can  only  promise  you  that  it  won’t  go 
away  -  it  will  occur  with  increasing  tempo  and  ferocity. 

In  response,  we  may  need  to  act  locally  -  within  our  own 
national  or  regional  borders  or  spheres  of  influence  -  but 
we  must  think  globally  -  and  we  must  be  continuously 
vigilant 

Range  Uniqueness 

In  the  past,  from  my  Pacific  experience.  I’ve  seen: 
Australian  platforms  and  weapons  systems  being  tested 
on  European  and  US  ranges;  and  US  and  European 
platforms  and  weapons  being  tested  on  the  Australian 
ranges.  Usually,  this  business  is  non-competitive.  We  go 
to  particular  ranges  -  not  principally  because  of  their 
investment  in  the  technology  of  test  -  but  because  they 
offer  some  environmental  or  topographical  advantage. 

And  this  is  where  a  global  investment  strategy  is  most 
needed  -  as  without  it  -  the  ranges  will  not  be  separately 
defendable  on  a  domestic-use-only  basis.  And  either  we 
stand  together  on  this  or  we  will  surely  fall  separately. 

A  Forecast  for  the  Pacific 

In  the  Pacific  region,  the  member  nations 
arguably  have  some  of  the  most  rapidly  expanding  and 
sometimes  volatile  economies  in  the  world.  But  these 
national  economies  are  becoming  increasingly  tenuous 
owing  to  the  increasing  reliance  on  high  technology 
systems  to  improve  and  maintain  the  prosperity,  security 
and  quality  of  life  of  each  nation's  citizens.  And  common 
to  all  these  high  technological  systems,  both  military  and 
civil,  are  the  devastating  consequences  if  they  fail  -  where 
loss  of  life,  property,  trust,  money  -  or  all  of  these  -  are 
inevitable  -  and  where  today's  errors  readily  become 
tomorrow's  headlines. 

The  presence  of  ITEA  in  the  Asia/Pacific  region 
therefore  heralds  a  new  opportunity  for  all  Pacific 
nations,  large  or  small,  to  understand  how  to  develop  and 
field  high  technology  systems  with  known  confidence  and 
with  predictable  consequences  -  by  replacing  the  proven 
failures  of  conventional  wisdom  with  the  lessons  shared 
and  learned  among  experienced  T&E  professionals. 


‘  See  http://www.telemetry.org  for  details  and  updates. 


The  Pacific  Chapters 

At  present,  two  Chapters  of  ITEA  are  already 
established  in  the  Asia/Pacific  region,  the  Southern  Cross 
Chapter  (in  Australia)  and  the  Mid-Pacific  Chapter  (in 
Kauai).  Their  demographics  are  illustrated  at  Figure  1. 

Figure  1  Pacific  Chanter  Demographics 
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The  oldest  of  these  Chapters,  the  Southern  Cross 
Chapter,  was  formed  in  1992  and  its  membership 
presently  spans  Australia  and  New  Zealand. 

As  a  service  to  the  T&E  community  in  the  Pacific 
Region,  the  Southern  Cross  Chapter  has  already  hosted 
eight  annual  conferences  on  T&E  —  with  the  year  2000 
event  to  be  in  Queensland  later  this  year.  And  last  year, 
the  Mid-Pacific  Chapter  hosted  an  enormously  successful 
Test  and  Training  Workshop.  I  am  confident  it  will  be  the 
first  of  many. 

Pacific  Region  -  Claims  to  Fame 

So  what  are  we  famous  for  in  the  Pacific  region 
apart  from  regularly  holding  T&E  professional  forums. 
Well  I  was  going  to  suggest  our  friendliness  -  but  I  then 
recalled  that  the  discoverer  of  Australia  was  an 
Englishman  by  the  name  of  Captain  James  Cook  and  that 
he  was  killed  in  the  Pacific  in  the  ‘Friendly  Isles’  -  which 
is  today  located  near  another  source  of  imfriendliness  - 
the  French  Pacific  nuclear  test  site. 

That  aside,  some  of  us  at  least  are  famous  for  our 
friendliness  -  so  come  on  down  to  our  T&E  Conference  in 
Queensland  later  this  year  to  judge  for  yourselves.  Known 
as  SETE-2000,  the  event  will  be  co-sponsored  by  the 
Southern  Cross  Chapter  of  ITEA  and  the  Systems 
Engineering  Society  of  Australia  [SESA]. 

Australian  Successes 

Other  claims  to  fame? 

Australian  Department  of  Defence 
Well  as  part  of  its  success  story  in  the  Australasian 
region,  the  Southern  Cross  Chapter  of  ITEA  has 
facilitated  the  development  of  Australian  Defence  Test 
and  Evaluation  Policy  and  acted  as  the  catalyst  for  the 
production  of  a  Directory  to  the  Australian  Defence- 
owned  Test  Ranges  and  Facilities  -  and  its  ultimate 
inclusion  in  the  International  directory. 

More  recently  (and  principally  through  its  Defence 
Science  members)  it  has  also  nurtured  the  establishment 
of  an  Australian  Defence  Simulation  Office  to  guide 
strategic  investment  in  Modeling  and  Simulation  [M&S]. 
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You  will  no  doubt  recognise  that  M&S  is  an 
important  component  of  the  T&E  tool-kit  -  as  it  embraces 
the  ‘prediction  tools’. 

What  remains  to  be  done  under  the  Australian 
T&E  policy  is  to  harmonise  this  with  the  investment 
planning  for  the  ‘observation  tools’  -  the  test  and  training 
ranges  and  facilities  -  which,  apart  from  their  traditional 
role  in  weapons  systems  testing  are  now  essential  to 
assess  the  fidelity  of  models  and  simulations. 

Research  and  Education 

The  Southern  Cross  Chapter  has  also  pioneered  the 
establishment  of  a  “regional”  centre  of  excellence  to 
facilitate  post-graduate  education,  training,  research  and 
technology  transfer  in  T&E.  Known  as  the  Australian 
Centre  for  Test  and  Evaluation  (ACTE)  and  located  at  the 
University  of  South  Australia,  ACTE  has  already 
achieved  considerable  international  respect.  The  Centre 
is  now  frequented  by  both  civil  and  military  researchers, 
students  and  staff  from:  the  Asia/Pacific  countries  (such 
as  India,  Korea,  and  Singapore)  from  Israel  and  from 
Europe;  and  it  has  also  surpassed  expectations  -  by 
attracting  collaborative  research  investment  from  beyond 
the  Asia/Pacific  region  -  such  as  the  United  States  and 
Europe. 

Perhaps  this  is  living  testimony  that  T&E  cannot 
be  departmentalised,  or  regionalised  -  as  it  is  tmly  global 
in  nature.  And  more  recently,  ACTE  has  expanded  to 
incorporate  the  education  and  research  needs  of  the 
Systems  Engineering  community  -  to  now  become  the 
Systems  Engineering  and  Evaluation  Centre  [SEEC]. 

This  initiative,  for  both  the  Systems  Engineering  and  the 
Test  and  Evaluation  communities  enables  further 
promotion  and  facilitation  of  constructive  dialogue  on 
their  intersecting  interests. 

Need  for  good  University  Links. 

Both  ITEA  and  SEEC/ACTE  are  success  stories. 
And  the  more  I  read  about  the  pioneers  in  this  exciting 
new  field  of  science  and  engineering,  the  more  I  believe 
we  should  celebrate  them  -  and  take  opportunity  to  join 
with  them  -  to  continuously  expand  and  improve  on  the 
foundations  they  have  already  laid. 

In  the  United  States,  I  know  there  is  good  dialogue 
occurring  on  T&E  with  a  number  of  Universities  and  their 
research  centres.  I’m  reluctant  to  name  them  for  the  fear 
of  leaving  one  out  -  but  perhaps  notable  is  the 
establishment  of  the  Test  and  Evaluation  Research  Centre 
[TEREC]  at  the  Georgia  Technical  Research  Institute 
[GTRI]  -  as  it  was  through  GTRI  that  the  Australian 
centre  got  up  and  running  and  we  are  indebted  to  the 
people  of  vision  there  who  made  this  possible.  And  I  bear 
personal  testimony  that  this  was  a  gracious  and  unselfish 
act  in  this  day  and  age  of  ‘every  man  for  himself. 

Likewise,  I  know  the  Mid-Pacific  Chapter  has 
strong  links  with  the  US  University  system  and  has 
notably  held  a  number  of  their  Chapter  events  with  the 
participation  of  the  High  Speed  Computer  Centre  at  the 
University  of  Hawaii. 

In  the  United  Kingdom,  I  understand  that  a 
nucleus  for  education  and  research  was  recently 
established  with  the  Defence  Engineering  and  Systems 
Group  of  University  College  London  -  so  please  support 


that  for  all  you  are  worth.  Because  if  you  are  like  us 
‘Down  Under’  -  the  pressures  of  modem  development 
programmes  leave  no  time  to  think  -  meaning  that  the 
whole  workforce  is  captive  to  the  ‘here  and  now’  with 
little  freedom  or  capacity  to  think  about  investment 
strategies  to  meet  the  demands  of  tomorrow. 

The  Pacific  Region  -  Major  Capabilities 

Before  exploring  an  investment  strategy  -  let’s 
look  at  some  specific  examples  of  what  we  are  famous 
for  in  the  Pacific  -  because  perceptions  can  be  deceiving. 
An  example  of  what  I  mean  by  this  is  evident  on  a  map 
produced  in  1996  by  National  Geographic  which  states 
some  quantitative  measures  for  the  investment  that  the 
United  States  Department  of  Defence  has  made  in  its 
military  reservations  and  test  and  training  ranges. 

The  map  witnesses  that  the  military  reservations  in 
the  US  occupy:  “27  million  acres  found  mainly  in  the 
South  and  West,  which  are  home  to  220  threatened  or 
endangered  species  and  contain  100,000  archaeological 
sites. 

Well  we  might  all  wonder  how  the  European  or 
the  Pacific  Region  investments  compare  to  that  -  but  I 
don't  think  any  of  us  in  Europe  or  the  Pacific  will  be 
trying  to  out-do  those  figures  as  an  exercise  in  one-up- 
man-ship. 

So  just  what  is  the  Pacific  Region  famous  for? 

Well  to  help  answer  that  question  I  asked  both  the 
Southern  Cross  Chapter  and  the  Mid-Pacific  Chapter  to 
provide  some  example  inputs  -  and  the  coverage  is  partly 
shown  at  Figure  2.  Kwajalein  gets  highlighted  because  its 
the  hardest  to  find  on  a  map. 


Figure  2  A  Section  of  the  Pacific  Region 


Marshall  Islands  -  Kwaialein  Missile  Range 
The  Marshall  Islands.  Population  43,380.  Home  to  the 
Kwajalein  Missile  Range.  Whaf  s  it  famous  for?  Mission 
support  for  theatre  ballistic  missile  and  intercept  testing; 
and  space  debris  monitoring  among  others, 

Hawaiian  Islands  -  Pacific  Missile  Range  Facility 
The  Hawaiian  Islands.  Population  1.1 8M.  Home  to  the 
Pacific  Missile  Range  Facility  [PMRF]  -  one  of  the 
largest  test  and  training  ranges  in  the  world.  What’s  it 
famous  for?  Multi-national  integrated  test  and  training.  Its 


^  National  Geographic  map.  Federal  Lands  and  the  Fifty 
States,  published  Oct  96 


IIL3 


area  of  coverage  is  illustrated  at  Figure  3  and  its 
connectivity  with  other  ranges  is  illustrated  at  Figure  4. 


Figure  3  -  PMRF  Area  of  Coverage 

With  reference  to  Figure  3,  PMRF  has  the  ability  to 
provide  simultaneous  real-time  tracking  information  on 
participants,  targets,  and  weapons  on  its  42,000  square 
miles  of  sea  and  airspace 


Figure  4.  Range  Cormectivity 
Australia  -  multiple  ranges. 

Australia.  Population  18.3M.  Home  to  multiple  air,  land 
and  sea  ranges  -  which  do  not  currently  have  central 
ownership.  In  this  scenario,  the  Australian  Director  of 
Trials^  in  conjunction  with  the  single  service  T&E 
principals  is  currently  planning  to  better  coordinate  T&E 
direction  and  resource  investment.  We  wish  them  well. 

Proof  and  Experimental  Range.  The  Army’s  Proof  and 
Experimental  Establishment  is  located  at  Port  Wakefield 
in  South  Australia  -  it  is  coastal  and  tidal  -  and  famous  for 
the  soft  recovery  of  projectiles. 


^  In  Australia,  the  Trials  Directorate  is  a  component  of 
the  Defence  Science  and  Technology  Organisation  and 
under  the  current  leadership  of  Colonel  Brian  Hall. 


Jervis  Bav  Range  Facility.  This  range  is  located  in  New 
South  Wales  and  provides  mission  support  for  drone 
engagements  and  through-life  telemetered  missile  firings 
for  our  Air  Force  and  Navy. 

The  Woomera  Range.  This  range  is  illustrated  at  Figure 
5.  The  South  Australian  component  of  this  occupies 
127,000  sq/km  and  enjoys  virtually  unencumbered 
airspace. 

Its  famous  for  its  umestricted  views  of  the  horizon, 
its  golf  course  -  voted  the  second  worst  in  the  world  -  and 
even  more  famous  of  late  because  there  is  not  enough 
domestic  work  to  keep  it  alive. 

Its  official  mission?  It’s  searching  for  one! 


Figure  5.  The  Woomera  Range 

New  Zealand.  Population.  3.5M.  Home  to  Industrial 
Research  Limited  -  where  the  privatisation  of  many 
traditionally  Government  research  activities  has  already 
occurred. 

Industrial  Research  Limited  HRLI.  What’s  it  famous  for? 
In  my  opinion,  and  as  an  encouragement  to  us  all  -  its 
resourcefulness  and  its  apparent  lack  of  organisational 
barriers  to  doing  things  productively. 

The  example  given  me,  which  was  a  maritime 
example,  testifies  to  this;  viz;  I  imagine  that  anyone  who 
has  a  professional  interest  in  T&E  has  read  the  US  DoD 
Simulation,  Test  and  Evaluation  Process  [STEP]. 

Well  it  seems  that  the  New  Zealanders  don’t  call  it 
STEP  -  they  call  it  “the  common  sense  process". 

The  process  they  followed  to  predict  and  measure 
the  dynamic  loads  on  a  surface  vessel  in  a  variety  of  sea 
states  almost  precisely  follows  STEP  and  yields  outcomes 
that  are  cmcial  to  understanding  a  vessel’s  seakeeping, 
stractural  integrity  and  human  comfort  factors. 

Their  ‘common  sense  process  ’  included  the 
development  of  a  finite  element  model,  a  prototype,  an 
installed  test,  and  a  simulation  that  was  followed  up  by 
sea  trials  to  assess  the  fidelity  of  the  predictions. 

The  trials  research  vessel,  illustrated  at  Figure  6 
was  ‘borrowed’  from  the  NZ  water  police. 

Resourcefulness?  Well  the  same  people  who  did 
this  work  for  the  NZ  maritime  community,  did  the  same 
for:  a  flutter  programme  on  an  A4  Skyhawk  for  their  Air 
Force  and  for  the  sea- worthiness  and  stmctural  integrity 
of  New  Zealand’s  contender  in  the  Americas  Cup. 
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Figure  6.  IRL-NZ  Trials  Research  Vessel 


I  also  understand  that  the  IRL  folks  spin-off  their 
experience  to  other  diverse  fields  ranging  from  human  hip 
prosthesis  to  the  manufacture  of  plastic  bags!  I  can’t 
speak  for  the  United  States  or  Europe,  hut  in  Australia,  I 
suspect  we’d  be  tripping  over  all  sorts  of  federal,  state 
and  industry  ‘research  fiefdoms’  if  we  tried  to  do  the 
same. 

And  on  that  note,  I  don’t  plan  to  speak  for  the 
European  or  the  US  or  the  Israeli  experience  of  T&E  but 
in  Australia,  national  leadership  on  T&E  is  currently 
conspicuous  by  its  absence.  And  everyone  it  seems  is 
expecting  Defence  to  take  leadership  on  the  issue. 

Ranee  Alliances  -  A  Compelling  Need. 

At  the  grass-roots  level,  there  is  current  talk  of 
forming  a  Pacific  Range  Alliance  that  would  give 
connectivity  and  load-sharing  between  the  major  Pacific 
Ranges  such  as  KMR  in  the  Marshall  Islands,  PMRF  in 
Kuaui  and  the  Woomera  range  in  South  Australia. 

I  dare  to  suggest  that  it  not  only  needs  to  be  talked 
about  -  it  needs  to  happen  quickly  -  and  it  needs  to  be  part 
of  something  even  bigger  which  will  enable  ‘plug’n’play’ 
of  all  range  participants  globally.  For  without  that  global 
vision,  I  believe  a  number  of  things  will  happen. 

•  Interoperability  testing  will  suffer  exquisitely.  [If  not 
addressed  it  means  that  the  next  generation  weapons 
systems  and  platforms  destined  for  coalition  force  or 
peace-keeping  operations  will  not  ‘plug’n’play’  and 
become  legacy  systems  before  they  are  ever  fielded.] 

•  Unique  range  capabilities  will  be  lost  forever  as  it  will 
be  unaffordable  to  recover  them,  [topography, 
environment  -  arctic,  desert,  tropical,  electromagnetic 
environment  etc] 

•  Every  range  participant  will  be  faced  with  unique 
spectrum  allocation  and  electromagnetic  environment 
incompatibility  problems  at  every  range  on  Earth. 

•  We  will  continue  to  be  ‘penny  wise  and  pound 
foolish’  -  as  currently  there  is  a  thousand-fold  more 
attention  being  paid  to  how  to  cut  salary  and  maintenance 
costs  at  the  ranges  compared  to  the  level  of  attention 
being  paid  to  optimising  their  ability  to  avoid  T&E 
progamme  costs. 

T&E  Economics  -  Some  Illustrations 

Let  me  illustrate  that  last  point  with  a  couple  of 
examples.  In  the  1990’s,  our  Air  Force  invested  in  a 


Transportable  Telemetry  System  at  a  cost  of  a  few  million 
dollars.  On  it’s  first  deployment  to  Woomera  it  detected 
problems  early  in  a  weapons  programme  -  which,  if  they 
had  remained  undetected  would  have  cost  the  Department 
of  Defence  twice  that  to  rectify.  It  paid  for  itself  on  its 
first  application  with  a  dividend  to  Defence  of  100%. 

So  what’s  happened  since  then?  No-one  has  kept 
any  records  of  the  cost  savings  or  cost  avoidances  that 
have  been  enjoyed  on  any  subsequent  programmes.  So, 
inevitably,  its  now  being  down-sized  and  possibly  out¬ 
sourced  to  minimise  staff  salaries  and  maintenance  costs  - 
in  an  absence  of  informed  understanding  of  how  this  will 
compromise  its  future  ‘cost-saving  or  cost-avoidance’ 
capacity. 

This  same  philosophy  applies  to  any  investment  in 
T&E  capabilities.  For  some  confirmation  on  this  I 
referred  to  a  DSTO  study  that  was  performed  on  the 
International  Follow-On  Structural  Test  Programme 
[IFOSTP]  for  the  F/A-18  fighter  aircraft.  The  cost  of 
investment  in  the  test  programme  was  $50M.  The  "cost 
avoidance'  that  this  investment  yielded  was  $3.5B. 
Similarly,  looking  at  the  sums  for  a  single  weapons 
clearance  programme:  the  investment  cost  was  $2.6M 
and  the  return  on  that  investment  (through  cost- 
avoidance)  was  $16.5M. 

Try  adding  up  those  investment  returns  for  every 
weapon  that  ever  needed  to  be  cleared  from  a  military 
weapons  platform! 

The  point  to  be  made  is  that  cutting  the  costs  of 
maintaining  the  T&E  capabilities  may  well  compromise 
their  potential  return  on  investment  quite  dramatically. 

Why  aren’t  we  looking  at  the  economics  of  T&E 
from  an  investment  strategy  perspective  so  that  we  can 
make  informed  decisions.  We  cannot  afford  to  be  ‘penny 
wise  and  pound  foolish’. 

Measuring  Range  Productivity 

Based  on  the  economic  rationale  explored,  I  would 
strongly  assert  that  the  principle  measure  of  productivity 
for  a  Test  Range  or  Facility  stands  on  one  of  the 
fundamental  precepts  of  T&E  -  to  detect  problems  while 
they’re  cheap  to  fix. 

Some  dispel  this  precept  as  technical  risk 
management  or  risk  mitigation  -  but  it’s  far  more  than 
that.  Its  an  economic  imperative  that  we  really  do 
understand  the  dividends  of  T&E  -  the  return  on 
investment. 

Everywhere  I  see,  the  cost  of  maintaining  the  test 
ranges  and  facilities  is  coming  under  intense  scratiny  - 
but  too  often  I  see  no  reference  at  all  to  the  economic 
return  being  enjoyed  because  of  their  existence.  This  is  a 
great  tragedy  and  one  that  compels  and  deserves  all  of  our 
attention  -  while  it’s  not  still  too  late. 

Need  for  a  Global  Investment  Strategy 

Well,  having  explored  what  some  of  us  may  be 
doing  wrong,  let’s  see  what  happens  when  we  do  things 
right.  What  would  be  the  expected  outcome  of  a  ‘global’ 
investment  strategy. 

•  global  range  connectivity  standards 

•  all  range  participants  ‘plug’n’play. 

•  seamless  interoperability  testing 
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•  ranges  build  on  their  strengths  and  not  their 
weaknesses 

•  distributed  subject  matter  experts 

•  cost-sharing  on  R&D  investments  in  next  generation 
range  technologies 

•  able  to  meet  surge  demands 

•  an  environment  that  will  inspire  bright  young  science 
and  engineering  graduates 

•  investment  economics  ‘understood’ 

•  cross-cultural  eiuichment 

Who  needs  to  consider  these  things?  Well,  as  a 
piece  of  personal  advice,  we  might  need  their  help  later  - 
but  we  should  not  start  with  our  politicans. 

As  an  example,  I  can’t  speak  for  the  behaviour  of 
the  US  Congress,  or  the  British  Parliament  or  the 
European  Union  -  but  recently,  in  Australia,  a  number  of 
High  School  girls  witnessed  a  sitting  of  the  Australian 
Parliament  and  were  interviewed  on  a  national  television 
transmission  immediately  afterwards. 

Asked  what  they  thought  of  the  experience,  one 
girl  replied,  'We//  if  we  behaved  like  that  we  ’d  all  be 
expelledl  ’ 

The  destiny  of  T&E  must  therefore  lie  in  our 
hands,  the  people  who  own  it,  practise  it  and  have 
responsibility  and  accountability  for  it.  And,  a  global 
T&E  investment  concept  that  promises  large  scale  cost 
avoidance,  interoperability,  connectivity  and 
compatibility  on  a  world  scale  must  be  addressed  urgently 
and  with  strategic  intent. 

We  need  to  reveal,  by  design,  the  chasm  between 
where  we  are  and  where  we  need  to  be  -  to  fuel  the  desire 
to  close  the  gap. 

We  need  to  understand  the  economics  of  ‘give  me 
a  million  dollars  to  spend  on  T&E  and  I  will  save  you 
two.’  And  like  the  outstanding  companies  in  industry,  we 
must  plan  to  get  to  the  future  first  -  to  create  the 
environment  in  which  the  future  can  flourish  -  safely  and 
economically.  To  do  this,  we  need  to  be  brave  and  not 
timid,  we  need  to  be  transformationary  not  evolutionary  , 
and  we  need  to  consider  leap-ahead  concepts  not  creep- 
ahead  concepts. 

And  as  a  final  flourish,  which  will  perhaps  put  my 
personal  credibility  on  the  line.  Give  me  a  million  dollars 
to  spend  on  T&E  and  I  will  promise  to  save  you  two  -  but 
I  will  expect  5%  of  the  dividend,  the  cost  avoidance  that  I 
achieve  for  you.  This  will  not  only  encourage  me  to  do 
even  better  next  time,  I  will  use  it  to  invest  in  my  own 
readiness  to  serve  your  future  needs. 

A  Recommendation 

I’d  like  to  make  a  call  to  everyone  in  the  T&E 
‘business’  to  consider  these  things.  In  the  current 
RMA/RMB  climate  we  need  to  enable  pur  decision¬ 
makers  in  Government  to  introduce  new  technologies  into 
civil  or  military  service  with  known  confidence  and 
predictable  consequences  —  and  to  leverage  the 
international  competitiveness  of  our  industries  by  helping 
them  to  place  'test-proven'  products  on  the  global 
supermarket  shelf. 

So  if  you  would  like  to  monitor  or  contribute  to 
what's  happening  in  T&E  in  your  region  please  contact 
your  local  Chapter  President.  If  you’d  like  to  start  your 


own  Chapter,  talk  to  your  Regional  Vice-President  or 
ITEA  HQ. 

For  if  we  continue  to  be  “survival  centred, 
maintenance  budget  based  and  competition  driven  ,  1 
believe  unique  capabilities  will  be  lost  forever  to  all  of 
humankind.  Instead,  I  believe  we  must  be  “vision  lead, 
capability  centred  and  cooperation  driven”  -  and  be 
passionate  about  getting  to  the  future  first. 

History  may  well  teach  us  that  “vision  without 
dollars  is  hallucination”  -  but  we  must  all  have  the 
product  before  we  can  sell  it  -  so  we  do  need  enough 
dollars  and  enough  commitment  to  manufacture  the 
vision  -  and  we  need  to  start  now. 

I  fully  recognise  that  today,  we  live  in  a  multi¬ 
cultural  society  but  as  some  evidence  of  divine 
precedence,  try  Proverbs  29:18;  'Without  vision  the 
people  perish." 

We  need  people  who  are  able  to  get  their  arms 
around  these  broad  concepts;  people  of  vision  who  think 
with  strategic  intent  and  in  systems-independent  terms 
and  at  systems-independent  levels  of  abstraction;  people 
who  know  what  the  science  and  technology  horizons  are; 
and  people  with  a  fire  in  their  belly  to  make  it  happen. 
And  in  responding  to  the  winds  of  change,  we  need  the 
leadership  of  the  future  and  not  the  studied  helplessness 
of  the  past, 

As  an  international  professional  society,  ITEA  has 
neither  the  authority  nor  the  funding  to  address  this  -  but 
I  strongly  believe  that  its  members  and  its  forams  are  the 
right  place  for  it  to  be  explored  and  discussed.  So  will 
everyone  please  treat  yourselves  to  establishing  this 
vision  -  for  I  believe  you're  worth  it. 
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Abstract 

Project  "Insight"  was  conducted  three  years 
ago  by  the  Defence  Evaluation  and 
Research  Agency  (DERA)  to  establish  the 
key  drivers  likely  to  influence  global  security 
in  the  period  2015  to  2040.  The  project  was 
initiated  under  the  auspices  of  the  UK 
Technology  Foresight  Defence  and 
Aerospace  Panel,  aiming  to  provide 
guidance  for  strategic  planning  in  defence. 

To  help  tackle  the  increased  uncertainty  and 
complexity  of  the  future,  the  project 
undertook  a  series  of  interviews  across  a 
wide  population  of  informed  people  around 
the  world.  A  small  number  of  focused  studies 
were  also  conducted.  A  substantial  period  of 
synthesis  engaged  recognised  defence  and 
scenario  planning  experts,  including  an 
international  symposium.  The  project 
produced  three  related  outputs: 

•  A  set  of  core  drivers  characterised  in  terms 
of  the  issues  they  raise,  the  outcomes  they 
imply  and  associated  indicators. 

•  A  range  of  contrasting  macro  scenarios 
illustrating  the  way  in  which  differing 
outcomes  for  the  set  of  drivers  might 
produce  qualitatively  different  international 
structures  and  relationships. 

•  A  proposal  for  an  evidence-based  hybrid 
planning  strategy  -  ‘the  Insight  method’  - 
encompassing  both  long  range  planning 
and  nearer  term  crisis  management. 

This  paper  outlines  the  Insight  method 
before  presenting  a  summary  of  the  ‘drivers’ 
aspect  of  the  work  and  their  implications. 
The  UK  Cabinet  Office’s  Strategic 
Chailenges  project,  run  by  its  Performance 
and  Innovation  Unit,  recently  drew  on  an 
enormous  range  of  futures  work,  including 
Project  Insight.  Its  consolidated  paper 
confirms  the  currency  of  the  Insight  work. 

1  -  Insight  method 

The  study  method  adopted  by  Insight  is  also 
the  basis  for  the  so-called  Insight  method,  as 
illustrated  in  Figure  1.  Briefly,  the  primary 
objective  of  the  Insight  method  is  to  establish 


a  framework  within  which  events  can  be 
more  effectively  mitigated  or  exploited.  This 
involves  responding  to  events  and  indicators 
reiated  to  previously  characterised 
outcomes,  optimising  process  and 
organisation  so  that  events  can  be 
responded  to  organically,  and  maintaining  a 
portfolio  of  investments  that  provide  as  broad 
a  coverage  of  possible  outcomes  as  is 
affordable. 


Figure  1  -  The  ‘Insight’  method 


2  -  ‘Insight’  drivers  of  global  security 

The  trends  and  events  were  characterised  to 
capture  knowledge  about  their  rate  of 
change,  plausible  divergencies,  and  stability 
in  the  context  of  wider  systems,  and  for 
convenience  were  grouped  under  the  eight 
driver  headings  in  Figure  2. 


Figure  2  -  The  ‘Insight’  core  drivers 


I  -  Demography 

Scope  of  driver:  This  driver  area  includes 
all  aspects  of  change  in  population  size  and 
composition.  In  many  areas,  indicators  such 
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as  age  and  size  of  population  will  be  well 
documented  in  government  censuses.  In 
unstable  regions  population  size  and 
composition  may  have  been  changed 
significantly  by  war,  famine  or  disease,  and 
accurate  figures  will  be  difficult  to  establish. 
Some  governments  may  also  publish 
misleading  figures.  Less  quantifiable 
indicators  should  also  be  used  to  assess 
projected  demographic  change;  for  example, 
increasing  literacy  rates,  particularly  among 
women,  can  be  expected  to  lead  to  declining 
birth  rates. 

Potential  key  security  issues 

•  Rapid  population  growth  is  leading  to  growing 
competition  for  resources,  mass  migration  and 
regional  instability,  as  well  as  pressures  on 
immigration  policies  of  the  richer  nations. 

•  The  aspiration  of  the  poorer  90%  to  live  like 
the  richer  10%  is  creating  new  demand  on 
natural  resources,  and  undermining  traditional 
communities. 

•  Growing  populations  of  unemployed  and 
disaffected  youth  in  developing  countries 
threaten  civil  stability  (the  potential  effect  of 
so-called  ‘testosterone  power’). 

•  Growth  in  poorer  countries,  in  terms  of  sheer 
numbers  and  economic  clout,  challenges 
economic  and  political  dominance  of  the  richer 
nations. 

•  Ageing  population  in  richer  countries,  and  their 
growing  needs,  will  increasingly  drain  public 
resources;  it  also  means  fewer  young  people, 
available  for,  or  willing  to  join,  armed  services. 

•  Increasing  gap  between  the  number  of 
engaged,  educated  and  wealthy  individuals 
and  the  poor  and  disengaged.  Many  nations 
will  have  more  people  than  they  need  for 
successful  economic  growth.  Those 
disengaged  and  without  purpose  could  be  an 
increasing  source  of  tension. 

II  -  Environmental  change 

Scope  of  driver:  This  driver  area  includes 
all  aspects  of  environmental  degradation  or 
improvement,  both  anthropogenic  and 
arising  from  natural  geological,  climatic  or 
other  causes.  Trends  might  include  air 
pollution  levels  arising  from  industrial  and 
domestic  use  of  fuel,  sea  pollution  levels  and 
their  influence  on  fish  stocks  and  human 
health,  carbon  dioxide  levels  in  the  upper 
atmosphere,  forest  clearance  and 
biodiversity.  Some  indicators  will  be 


quantifiable  -  measurement  of  water  tables 
or  of  airborne  and  waterborne  pollution  over 
time,  for  example.  Projected  environmental 
change  should  also  be  assessed  using 
indicators  such  as  public  attitudes  to  car  use 
or  awareness  of  industrial  pollution,  and  the 
degree  of  international  political  co-operation 
over  matters  such  as  anthropogenic  climate 
change. 

Potential  key  security  issues 

•  To  counter  effectively  the  growing  threat  of 
global  climate  change  we  require  trans- 
nationai  consensus  and  robust  globai 
institutions  that  do  not  yet  exist,  it  is  a  problem 
that  cannot  be  solved  unilaterally.  How  do  we 
persuade  nations  to  deveiop  sustainably,  and 
persuade  developed  nations  to  reduce 
consumption? 

•  Access  to  resources  such  as  energy  and 
minerals  continues  to  be  a  source  of  tension. 
This  may  increase  as  more  countries 
industrialise  and  populations  and  expectations 
increase. 

•  Ciimate  change  and  migration  patterns  could 
shift  patterns  of  disease  distribution. 

•  Growing  populations  are  piacing  increasing 
pressure  on  key  resources  such  as  water  and 
productive  agriculturai  land,  which  can  be  the 
spur  of  both  ethnic  and  inter-state  violence, 
with  international  ramifications. 

•  Forms  of  poliution  such  as  nuclear  emissions 
are  not  constrained  by  state  borders  and  can 
exacerbate  tensions  between  polluting  nations 
and  their  neighbours,  inciting  retaliation. 

•  Large  numbers  of  ‘environmentai  refugees’  are 
possible  (300-500  million  by  around  2000), 
resuiting  in  regional  destabilisation. 

•  The  aspiration  of  the  deveioping  world  for 
higher  standards  of  living  places 
unprecedented  pressure  on  the  global 
environment  and  resources,  which  have  until 
now  been  monopolised  by  the  West.  How  can 
this  problem  be  managed? 

•  Will  the  constraints  of  pollution  enforce  a 
‘wartime  economy’  of  ciose  reguiation  and 
rationing  of  damaging  agents  and  processes? 

•  How  do  we  influence  nations  whose  emissions 
may  degrade  the  local  or  global  environment? 

•  Many  in  the  population  increasingly  call  out  for 
action  on  environmental  issues.  If 
governments  do  not  act  effectively  then  how 
may  people  seek  to  solve  this  problem? 
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•  Use  of  environmental  damage  as  a  tool  in 
conflict,  for  example,  the  Iraqi  burning  of  oil 
wells  and  the  release  of  oil  in  the  Gulf,  and  the 
draining  of  Marshlands  in  S  Iraq,  use  of 
defoliants  by  the  US  in  Vietnam.  Aggressive 
actors  may  exploit  environmental 
vulnerabilities  or  use  them  to  coerce 
opponents. 

Ill  -  Globalisation  of  economic  systems 

Scope  of  driver:  This  driver  area 
enconnpasses  aspects  of  all  economic 
change,  but  particularly  focuses  on  the 
growing  interdependence  of  economic 
systems  within  regions  and  around  the  world. 
The  direction  of  capital  flows  and 
international  investment  as  an  indicator  of 
current  and  potential  economic  growth 
should  be  measured,  as  should  government 
legislation  such  as  the  reduction  or  increase 
of  tariff  barriers  and  limits  to  international 
trade  and  foreign  direct  investment. 
Interdependence  is  increasingly  enhanced 
by  the  creation  of  economic  areas  such  as 
the  European  Union  and  the  Association  of 
South  East  Asian  Nations,  and  the 
development  of  these  should  be  monitored. 
Economic  globalisation  may  lead  to  the 
marginalisation  of  less  economically  dynamic 
individuals  as  well  as  states  or  whole  regions 
from  the  global  economic  network.  Evidence 
of  this  marginalisation,  leading  to  resentment 
and  instability  within  and  between  states, 
should  be  sought.  Other  indicators  of 
anticipated  economic  change  might  include 
government  policies  to  redistribute  wealth, 
education  levels,  unemployment,  and 
environmental  pollution  -  which  have  the 
potential  severely  to  disrupt  regional  and 
national  economic  systems. 

Potential  key  security  issues 

•  Governments  are  steadily  losing  control  over 
their  economies  to  international  institutions  or 
the  organic  ‘hand  of  Adam  Smith’,  with  all  that 
implies  for  reductions  in  their  power  and 
influence. 

•  Shifts  in  the  location  of  economic  power  may 
generate  shifts  in  the  location  of  diplomatic 
and  military  power,  with  unpredictable  political 
ramifications  as  ‘old’  nations  readjust. 

•  A  growing  proportion  of  mid-ability  people  in 
the  developed  nations  will  not  offer  a  globally 
competitive  level  of  ‘worth’  commensurate  with 
acceptable  income.  Disenchanted  under¬ 
classes  may  become  an  endemic  cause  of 
internal  conflict  and  breeding  ground  for  trans¬ 
national  aggression. 


•  The  strength  of  national  economies  affects 
defence  spending,  e.g.  the  West  experiencing 
a  decline,  mirrored  by  a  rise  in  the  East. 

•  With  growing  global  competition,  and  smaller 
internal  markets,  ability  to  maintain  a  national 
defence  industry  is  becoming  harder. 

•  The  globalisation  of  the  arms  industry  leads  to 
a  proliferation  of  both  weapons  and  weapon¬ 
making  technologies  among  nations  that  are 
or  might  one  day  become  adversaries. 

•  Growing  economic  interaction  and  the  rise  of 
‘interdependence’  between  nations  may 
improve  relations  or  foster  mercantilistic 
competitions. 

•  Current  economic  trends  are,  in  some  areas, 
leading  to  a  growing  gap  between  ‘haves’  and 
‘have  nots’  and  a  decline  in  social  cohesion 
and  stability  both  within  nations  and  between 
nations, 

IV  -  Coherence  of  politics 

Scope  of  driver:  This  driver  area  spans  the 
range  of  factors  that  impinge  upon  the 
strength  and  iegitimacy  of  political  power,  the 
form  and  impact  of  formai  politicai  process 
and  the  circumstances  that  determine  its 
nature  within  states.  The  strength  and 
stabiiity  of  politicai  regimes  should  be 
measured  through  the  analysis  of  the 
support  bases  of  political  leaders  within  the 
domestic  polity  and  in  the  population  of  the 
country.  The  international  status  of  the 
regime  also  needs  to  be  considered  in  this 
driver  area,  as  international  condemnation  of 
a  regime  may  have  a  significant  impact  on  its 
stability  at  home. 

Important  domestic  indicators  of  political 
strength  will  be  the  level  of  popular 
involvement  in  the  political  process,  the 
state’s  human  rights  record,  the  level  of 
violence  surrounding  electorai  processes 
and  civiiian  control  of  the  armed  forces.  The 
emergence  of  alternatives  to  the  leadership 
within  or  outside  the  government,  perhaps  of 
ethnic  groups,  should  be  considered. 

Although  the  jurisdiction  of  a  regime  may  be 
recognised  internationally,  this  may  not  be 
reflected  in  the  actual  level  of  political  control 
that  a  government  is  able  to  exert  over  its 
territory;  the  existence  of  the  rule  of  law 
throughout  the  territory  is  an  important 
indicator  of  a  government’s  effectiveness. 
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Potential  key  security  issues 

•  The  type  and  nature  of  national  leaders,  their 
intent  and  degree  of  public  support. 

•  Declining  confidence  of  populations  in 
governments  and  inhomogenities  in  support 
threaten  legitimacy  and  the  government’s 
power  to  act  in  the  face  of  perceived  threats, 
internal  or  external. 

•  Hesitancy  gripping  the  latest  wave  of 
democratisation  in  the  developing  countries 
and  the  former  Soviet  Union  raises  spectre  of 
tumult  and  unpredictable  politics. 

•  Generalised  access  to  knowledge  and 
technology  outside  the  sphere  of  state 
influences  have  empowered  a  range  of 
hitherto  ‘hidden’  actors. 

•  The  increasing  number  of  actors  in  the 
international  arena,  combined  with  the  growing 
number  of  issues  that  motivate  them  and  the 
means  at  their  disposal  to  pursue  their 
objectives,  has  greatly  increased  the 
complexity  of  global  politics. 

•  The  shift  from  state-centric  sources  of  power 
to  a  range  of  non-state  actors  has  resulted  in 
much  less  predictability  and  greater 
turbulence. 

•  Traditional  criminal  groups  -  including 
terrorists,  and  drug  and  arms  traffickers  -  have 
greatly  increased  their  effectiveness  and 
legitimacy  through  the  extension  of  their 
international  networks. 

•  The  growing  number  of  failed  states  means 
that  the  traditional  interlocutors  of  international 
agencies  and  governments  in  the  developing 
world  are  losing  their  legitimacy  and 
capabilities,  in  some  cases  resulting  in  power 
voids. 

V  -  Alliances  and  alignments 

Scope  of  driver:  The  character  and  strength 
of  international  alliances  and  alignments  - 
both  formal  and  informal  -  is  assessed  in  this 
driver  area.  The  continuing  existence,  role 
and  status  of  the  North  Atlantic  Alliance  and 
the  nature  of  the  transatlantic  relationship  will 
be  important  indicators  of  change  to  the 
British  and  European  security  environment. 
The  emergence  of  security  alliances  from 
regional  economic  and  political  groupings 
such  as  the  Association  of  South  East  Asian 
Nations  and  the  Organisation  for  African 
Unity  will  be  important  to  monitor.  Military 
coalitions,  including  peace  support 
operations,  are  included  in  this  driver  area. 


The  legitimacy  of  such  coalitions  is 
increasingly  established  through  United 
Nations  Security  Council  Resolution: 
therefore  the  status  and  effectiveness  of  the 
UN  are  important  determinants  of  the 
coherence  of  these  alliances.  The  state  of 
the  UN  is  also  one  of  the  more  concrete 
indicators  of  the  existence  of  an  ‘international 
community’  in  any  meaningful  sense.  The 
development  of  the  Organisation  for  Security 
and  Co-operation  in  Europe  will  be  important 
to  assess. 

Informal  alignments  between  states  and  the 
emergence  of  dominant  regional  powers  will 
also  be  included  in  this  driver  area;  for 
example,  analysts  may  monitor  tensions 
between  the  governments  of  the  European 
Union,  or  look  for  indicators  of  Chinese 
ambition  to  extend  its  influence  over  political 
affairs  in  South  East  Asia.  Indicators  of 
change  in  this  driver  area  will  be  the 
establishment  of  international  treaties  and 
agreements,  the  strength  and  effectiveness 
of  regional  organisations,  patterns  of  trade 
(particularly  the  arms  trade)  and,  importantly, 
personal  relationships  between  heads  of 
government. 

Potential  key  security  issues 

•  The  character  and  strength  of  international 
alignments  (broadly  defined)  and  formal,  inter¬ 
state  security  alliances,  contribute  directly  to 
the  security  of  the  international  political 
system.  In  the  absence  of  unequivocal, 
external  threats,  which  have  a  unifying  effect, 
formal  security  alliances  may  become  more 
difficult  to  define  and  sustain.  Instead, 
international  security  may  become  a  function 
of  ad  hoc  arrangements  and  ‘virtual  security 
alliances’. 

•  Identification  of  a  potential  adversary  has 
obvious  implications  for  the  general  military 
posture  to  be  adopted,  on  the  procurement  of 
military  equipment,  and  on  other  areas  such 
as  military  training,  reserves  and  force 
generation.  Formal  security  alliances  are 
called  into  question  because  of  the  diminution 
of  military  threat  and  because,  without  a  ‘clear 
and  present  danger’,  domestic  defence 
budgets  are  reducing.  In  these  circumstances, 
it  is  difficult  to  see  how  formal  security 
alliances  could  be  replaced  by  self-contained 
national  forces,  other  than  by  very  large 
powers.  So  it  must  be  asked  both  why  and 
how  states  will  commit  themselves  to  formal 
security  alliances. 

•  Security  alliances  formed  to  meet  a  crisis  or 
during  a  conflict  will  require  some  form  of 
legitimacy.  This  may  be  established  by  the 
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UN.  Where  the  establishment  of  legitimacy  is 
neither  automatic  nor  universally  accepted,  the 
result  may  be  that  ad  hoc  groupings  (i.e.  less 
formal,  less  durable  security  alliances)  are 
formed. 

•  What  role  does  a  state  wish  to  play  in  the  post- 
Cold  War  security  environment?  Is  the 
motivation  to  be  simply  national  defence  and 
security,  or  the  creation  and  maintenance  of  a 
global  security  framework?  An  isolationist  or 
unilateralist  trend  among  the  richer  and 
militarily  more  capable  states  will  be 
increasingly  Incompatible  with  the  co-operative 
approach  to  both  national  and  international 
security  that  emerging  global  problems  will 
require. 

•  Effective  security  alliances  should  not  be 
assumed  to  be  a  western/northern 
phenomenon;  as  developing  states  become 
more  powerful  and  become  more  organised 
regionally  and  by  sector,  they  may  be  able 
more  effectively  to  seek  their  individual  and 
collective  goals,  and  protect  their  interests. 

VI  -  Access  to  knowledge 

Scope  of  driver:  This  driver  area  covers  all 
aspects  of  the  explosion  in  access  to 
information  that  have  transcended 

geography  and  transformed  many  aspects  of 
political,  social  and  economic  life  in  the 
twentieth  century.  All  forms  of 

communication  and  education  should  be 
examined,  from  Internet  use  to  literacy  rates 
and  availability  of  basic  schooling. 

Control  of  information  is  increasingly  difficult 
for  governments  to  achieve,  and  it  will  be 
important  to  monitor  how  individuals  and 
small  groups  are  using  the  widespread 
availability  of  cheap  communications  to 
further  their  political  ends.  The  power  of  the 
media  can  be  enormous,  so  ownership  of  the 
media  is  of  crucial  importance.  The  effects  of 
broadcasting  on  public  attitudes  to  all  kinds 
of  issues  -  from  government  policies  to 
marriage  to  fashions  in  clothes  and  food  - 
may  be  difficult  to  predict  and  should  be 
monitored  carefully.  Changing  levels  of 
television  and  radio  use,  newspaper 
readership,  literacy  rates  and  the  availability 
of  public  and  private  transport  are  useful 
indicators  of  change  in  this  driver  area. 

However,  the  effects  of  increased  access  to 
knowledge  will  often  be  less  quantifiable,  as 
individuals’  reactions  -  even  to  the  same 
media  message  -  vary  enormously.  Perhaps 
more  important,  therefore,  may  be  analyses 
of  the  effects  of  changing  patterns  of 


information  availability  by  regional  experts, 
who  should  have  more  detailed 
understanding  of  the  society  in  question  and 
language  fluency. 

Potential  key  security  issues 

•  Access  to  knowledge  is  changing  the  balance 
of  power  between  individuals,  non-state  actors 
and  the  state,  leading  to  more  pluralist-based 
decision-making  processes  with  a  concomitant 
decline  in  the  ability  to  react  quickly  and 
decisively  in  emergency  situations. 

•  The  media  are  having  a  growing  impact  on 
both  government  actions  and  public 
perceptions  of  the  way  their  governments 
work,  with  a  growing  ability  to  influence  policy. 

•  Cultural  globalisation  as  driven  by  the  media  is 
creating  expectations  in  poor  countries  of  a 
kind  and  quality  of  lifestyle  that  may  not  be 
attainable.  This  may  cause  uncontrollable 
tensions  within  under-developed  countries. 

•  The  global  media  and  information  technology 
have  put  a  powerful  new  tool  In  the  hands  of 
terrorists  and  other  pressure  groups,  one  that 
can  both  ensure  anonymity  or  publicise  their 
activities  to  a  global  audience. 

•  Governments  will  lose  the  ability  to  control 
availability  of  information  and  set  chosen 
internal  agenda. 

•  More  information  wili  be  disseminated  via  ‘info- 
suck’  rather  than  edited  media.  This  will  have 
a  wide  range  of  implications,  especially 
increasing  the  power  of  the  capable. 

•  Validation  of  information  will  be  a  serious 
issue.  Organisations  will  be  able  to  endorse 
veracity,  or  that  have  a  reputation  for 
accuracy,  will  gain  in  influence. 

•  Global  movements  of  information  and  E- 
money  will  subvert  legal  controls,  creating 
uncontrolled  markets  for  information  services. 

•  As  the  reliance  on  information  technologies  for 
economic  power  grows,  the  premium  on 
protecting  information  systems  is  aiso 
increasing,  though  this  may  not  be  possible  to 
do  comprehensively. 

•  Access  to  knowledge  will  facilitate  the 
proliferation  of  skill  required  to  produce 
weapons,  especially  weapons  of  mass 
destruction. 

•  Effective  husbanding  and  exploitation  of 
knowiedge  will  become  the  primary 
discriminator  in  business  success,  requiring  a 
big  adjustment  to  balance  sheet-based 


1.5 


accounting  principles.  This  ‘whole  new  ball 
game’  of  info-assets  could  have  destabilising 
influences. 

VII  -  Science  and  technology 

Scope  of  driver:  This  driver  area  covers  ail 
technological  innovations,  the  implications  of 
their  development,  and  their  influence  on  the 
security  environment.  Although  technology  in 
warfare  is  an  obvious  example  of  how  this 
driver  influences  the  security  environment, 
military  technology  is  only  one  aspect  of  this 
broad  area. 

Developments  in  medical  science  may  have 
an  important  impact  on  population  size  and 
public  attitudes.  Communications 
technologies  are  likely  to  continue  to  grow, 
with  diverse  consequences.  Genetic 
engineering  may  provide  benefits  through 
increased  crop  yields  or  the  elimination  of 
hereditary  birth  defects;  but  they  may  wreak 
unexpected  havoc.  It  will  be  important  to 
monitor  not  only  scientific  breakthroughs  but 
also  their  (often  surprising)  impacts  on 
society. 

Potential  key  security  issues 

•  Technological  innovation  continues  apace, 
with  developments  which  can  disturb  the 
balance  between  aggression  and  mitigation, 
trigger  social  change,  and  pose  fundamental 
challenges  to  legal  frameworks. 

•  Barriers  to  exploiting  technology  are  reducing, 
and  the  traditional  monopoly  of  the  richer 
countries  on  research  and  development  is 
eroding. 

•  The  shift  in  balance  between  civil  and  military 
research  makes  proliferation  of  technologies 
imparting  military  superiority  much  harder  to 
control:  technological  edge  in  a  specific 
chosen  area  of  capability  may  be  available  at 
modest  cost,  even  to  non-state  actors. 

•  We  are  witnessing  enormously  rapid  progress 
in  understanding  of  genetics  and  of  the  nature 
of  the  human  mind;  this  may  unlock  a 
Pandora’s  box  of  technological  innovation  that 
could  impact  strongiy  on  society  and  the 
conduct  of  warfare. 

•  Much  current  development  gives  emphasis  to 
the  ability  to  design  and  adapt  products  in  a 
very  short  time  with  extensive  computer 
simulation  prior  to  prototype  construction;  this 
has  potential  to  transform  the  product  lifecycle, 
especially  in  the  military  sector. 


•  Scientific  ignorance  and  an  emphasis  on  the 
‘curative’  powers  of  technology  have  spawned 
an  optimistic  culture  that  neglects  efforts  to 
anticipate  problems;  this  approach  may 
founder  during  the  Insight  time  window. 

VIII  -  Public  perceptions  and  attitudes 

Scope  of  driver:  This  is  perhaps  the 
broadest  and  most  nebulous,  but  also  the 
most  important,  of  the  drivers.  It  covers 
change  in  popular  beliefs  about  a  whole 
range  of  issues,  from  attitudes  to 
government  taxation  to  perceptions  of 
foreigners  or  ethnic  minorities  to  religious 
beliefs. 

Opinion  polls  are  the  most  obvious,  although 
not  perhaps  the  most  effective,  indicators  of 
public  attitudes  -  the  answers  given  depend 
on  the  questions  asked,  and  may  reveal 
what  the  respondents  merely  think  they 
ought  to  say.  Other  indicators  of  changing 
public  attitudes  might  be  varying  levels  of 
participation  in  religious  activities,  ad  hoc 
political  protests  and  social  movements,  and 
changing  family  structures.  Like  the  ‘Access 
to  knowledge’  driver,  often  the  most  effective 
indicators  of  changing  trends  in  public 
perceptions  and  attitudes  will  be  provided  by 
regional  experts,  who  should  have  more 
detailed  understanding  and  language  fluency 
of  the  society  in  question. 

Potential  key  security  issues 

•  The  growth  of  trans-national  groupings, 
reinforced  by  globalisation  of  knowledge,  may 
create  new  axes  for  the  development  of 
perceptions  and  attitudes,  driven  by 
fundamentalisms,  professional  groupings, 
sectional  interests  and  the  media. 

•  Trends  toward  public  ‘insularisation’  in  the  face 
of  less  visible  threats  make  it  difficult  for 
governments  to  target  public  resources  to 
counter  them.  Armed  forces  need  to  think 
carefully  how  to  educate  peoples  to 
understand  their  role  and  relevance. 

•  The  heightened  sensitivity  to  casualties  in  the 
richer  nations  makes  public  support  for  military 
action  difficult,  and  this  may  impose  serious 
constraints  on  doctrine  and  tactics. 

•  The  growth  of  religious  fundamentalism  has 
been  a  feature  of  the  recent  past.  Will  this 
trend  continue  and  couple  more  strongly  into 
the  wider  security  scene,  or  will  other  effects 
such  as  globalisation  and  increasing  wealth 
drive  in  the  opposite  direction? 
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Instruments  of  security 


•  The  freedom  to  act  on  a  wide  range  of  issues 
is  constrained  by  public  reaction,  which  has 
caused  major  shifts  in  the  actions  of 
governments,  muiti-national  companies  and 
other  institutions. 

•  Managing  pubiic  attitude,  e.g.  through 
information  warfare,  may  become  a  criticai 
defence  /  security  capability. 

•  The  attitude  of  short-termism  and  defeatism 
prevalent  in  the  richer  countries  regarding 
many  global  security  problems  deflects 
attention  from  underlying  causes  and  possible 
preventive  actions. 

3  -  Implications  for  defence  capabilities 

Offerings  from  armed  forces 

The  analysis  of  trends  and  scenarios 
suggested  the  need  to  look  more  openly  at 
the  ‘offerings’  that  defence  provides  to  its 
stakeholders.  These  are  presented  under 
two  headings;  Defence  missions  (Figure  3) 
and  Diplomatic  missions  (Figure  4). 


Through  brainstorming  the  kinds  of 
capabiiities  and  instruments  of  security  that 
might  be  required,  it  is  possible  to  produce  a 
list  of  war-fighting  and  architectural 
capabilities,  as  shown  in  Figures  5  and  6 
respectively. 

Instruments  of  security 

Warfighting  capabilities: 

•  ability  to  threaten  and  coerce 

•  battlespace  superiority 

•  mobility  and  sustainability 

•  joint  readiness 

•  destruction  of  assets  and  infrastructure 

•  tactical  RISTA  and  operational  planning 

•  holding  ground 

•  close-in  and  special  forces 

•  counter  WMD 

•  Information  warfare,  psyops 

•  constabulary  force 


Figure  5  -  Warfighting  capabiiities 


Instruments  of  security 


Offerings  from  armed  forces 

Defence  mission  types: 

•  defend  attack  against  homeland 

•  support  dependent  territory  or  treaty  obligation 

•  succeed  in  regional  interventions 

>  coercion  to  mitigate  aggressive  threats 

•  protect  (or  exploit)  physical,  economic 
or  information  infrastructure 

•  support  to  civil  power 

PgRA 

Figure  3  -  Defence  mission  types 


Architectural  capabilities: 

•  presence  and  international  standing 

•  ability  to  marshal  military,  economic  and  diplomatic 
levers  to  achieve  a  result 

•  deterrence  through  commitment  to  timely  and 
proportionate  action 

•  intelligence 

•  intelligence  on  potential  actors 

•  planning  and  rehearsing  missions  , 

•;  re-establishing  governance, and  infirasthipture 


Figure  6  -  Architectural  capabiiities 


Capability  portfolio  analysis 


Offerings  from  armed  forces 


Diplomatic  mission  types; 

•  promote  state's  influence  and  signal  commitment 

•  intelligence  and  information 

•  protecting  trade  and  economic  interests 

•  deterrence  and  denial  of  opportunity 

•  peace  keeping  /  enforcing 


Figure  4  -  Dipiomatic  mission  types 


On  this  basis  it  is  possible  to  do  a  high-level 
capability  assessment,  such  as  the  one 
illustrated  in  Figure  7.  In  this  example, 
warfighting  capabilities  are  analysed  against 
each  of  the  four  macro  scenarios  (outlined  in 
Appendix  A)  and  the  issues  addressed, 
asking  how  we  think  current  plans  stack  up, 
whether  they  are  appropriate,  should 
investment  be  increased  or  decreased,  et 
cetera.  Although  the  example  given  is  purely 
illustrative  it  is  possible,  say,  that  in  three  of 
the  four  macro  scenarios  ‘destruction  of 
physical  infrastructure’  may  not  be  as 
relevant  as  at  present  and  that  current  plans 
may  be  quite  well  aligned  to  that  type  of 
situation. 
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Ulustrative  capability  portfolios 
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Figure  7 -Portfolio  analysis  -  warfighting 

As  far  as  architectural  capabilities  are 
concerned,  the  main  point  to  be  taken  from 
the  illustrative  diagram  on  the  left  is  that  an 
abundance  of  upward  arrows  implies  the 
need  for  a  shift  in  balance  of  investment. 
Warfighting  capabilities  are  still  important, 
but  we  need  to  catch  up  with  the 
architectural  capabilities. 


Illustrative  capability  portfolios 
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Figure  8  -  Portfolio  analysis-  architecture 


4  -  Summary  of  security  implications 

This  section  summarises  some  implications 
of  the  preceding  analysis  for  the  key  issues 
of  responsiveness,  warfare  and  acquisition. 

Responsiveness  as  a  military  capability 

The  inherent  complexity  and  uncertainty  of 
the  post-Cold  War  security  environment 
place  a  premium  on  the  responsiveness  of 
the  armed  forces  as  a  military  capability.  It  is 
vital  that  we  foster  agility  in  our  processes, 
including  the  ability  to  adapt  rapidly  and 
apply  existing  doctrine  and  tactics,  training 
and  readiness,  precision  logistics  and 
sustainability,  and  mission  planning  and 
control,  as  well  as  reducing  the  times 
involved  in  procurement,  design  and 
manufacture.  Above  all,  ‘plasticity’  -  the 


ability  to  adapt  design  and  efficiently  produce 
equipment  to  effectively  meet  changing 
requirements  -  and  plasticity  in  one’s 
systems  offer  a  decisive  advantage  -  in  being 
able  to  delay  procurement  and  keep  open  a 
wider  range  of  options  for  longer. 

Implications  for  responsiveness 

At  the  same  time  we  must  seek  to  improve 
current  approaches  to  crisis-management  - 
for  example,  by  developing  processes  that 
can  reliably  synthesise  plausible  future 
scenarios  and  provide  a  means  for 
recognising  and  monitoring  the  indicators 
and  signals  associated  with  them.  The 
process  should  be  methodologically  sound, 
unconstrained  to  the  extent  that  it  is  free  to 
step  outside  and  challenge  traditional 
thinking,  and  utilise  both  open  and  closed 
sources  of  intelligence  information.  Such  a 
process  is  currently  being  developed  by  the 
DERA  Project  ‘Insight’  team  in  collaboration 
with  the  Policy  Planning  Staff  of  the  Foreign 
and  Commonwealth  Office. 

Implications  for  warfare 

A  range  of  implications  for  warfare  stems 
from  the  greater  number  and  diversity  of 
actors  seeking  to  shape  and  shove  the 
course  of  global  security,  both  now  and  in 
the  future.  It  is  clear  that  reliable  intelligence 
and  surveillance  will  become  extremely 
valuable  assets,  and  that  intelligence 
information  in  the  future  will  require 
knowledge  not  only  on  technological  issues 
but  also  increasingly  on  the  understanding  of 
human  behaviours  and  intentions.  Increasing 
popular  access  to  high-impact  technologies 
such  as  the  internet  and  global  media  has 
made  it  possible  for  ordinary  individuals  and 
groups  to  direct  forms  of  persuasion, 
coercion  and  propaganda  at  other  individuals 
and  groups  using  means  previously  only  at 
the  disposal  of  states.  Dealing  with  these 
new  forms  of  potential  threat  implies  a 
complex  blend  of  military,  diplomatic  and 
economic  actions  by  the  state,  with  public 
opinion  acting  as  the  principal  moderator  of 
its  behaviour.  An  adverse  risk-reward  ratio 
could  result  from  the  adoption  of  certain 
actions  or  policies  by  states,  for  example  the 
targeting  of  charismatic  leaders  or  gurus. 

Increasingly,  the  integrated  architecture  of 
warfare  will  determine  the  overall  operational 
effectiveness  of  a  nation’s  forces,  rather  than 
the  functioning  of  stand-alone  weapons  and 
capabilities.  In  an  environment  where  total 
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knowledge  and  co-ordination  -  in  short, 
omniscience  -  is  tantamount  to  being  a 
decisive  weapon,  information  warfare, 
jamming  and  perception  manipulation  will 
likely  become  the  routine  modes  of  conflict. 
There  may  also  be  a  considerable  blurring 
between  what  we  mean  by  the  ‘teeth’  and 
what  we  mean  by  the  ‘tail’,  as  the  integrity  of 
‘soft’  background  logistics  and  analytical 
support  functions  becomes  crucial  to 
ensuring  the  effectiveness  of  more  traditional 
‘hard’  kill  components.  In  sum,  future  mission 
success  is  likely  to  be  driven  as  much  by  the 
strength  of  the  integrated,  architectural  ‘glue’ 
as  by  the  effectiveness  of  war-fighting 
capabilities  in  isolation. 

The  ‘Web  of  nations’  and  ‘Fragmented’ 
scenarios  reveal  the  importance  of 
maintaining  joint  training  and  readiness  in 
anticipation  of  the  need  to  form  rapidly  ad 
hoc  partnerships  and  alliances.  Role  sharing 
remains  a  possibility  in  the  ‘Subsidiarity’  and 
‘Blocs’  scenarios,  but  is  not  a  credible 
prospect  in  the  ‘Web’  and  ‘Fragmented’ 
worlds. 

Implications  for  acquisition 

Perhaps  the  central  issue  in  terms  of 
acquisition  remains  how  the  process  of 
defence  industry  consolidation  will  play  out 
and  what  implications  follow  from  this  for  the 
state  of  the  European  and  UK  defence 
industries.  It  is  possible  that  the  industry  is 
fast  approaching  a  ‘flip’  point,  as  the 
disengagement  of  governments  from  their 
client  suppliers  leaves  national  research 
expertise  to  find  its  own  way  on  the  world 
market,  and  the  ongoing  consolidation  of  the 
industry  world-wide  creates  a  small  number 
of  truly  global  suppliers.  The  virtual  versus 
vertical  manufacturer  structure,  significant 
advances  in  flexible  design  /  manufacture 
and  the  commodity  trading  of  defence 
equipments  all  represent  a  significant  shift 
away  from  traditional  practices  and  could 
well  become  a  reality  within  the  next  decade. 

Industry  must  also  come  to  terms  with  the 
increasing  commoditisation  of  defence 
products  and  services,  as  governments  shift 
from  traditional  processes  of  procurement  to 
favouring  off-the-shelf  limited  developments 
and  technology  insertion.  At  the  same  time 
the  real  military  benefits  of  rapid 
procurement  (that  the  short-term  and  low  risk 
procurement  process  preserves  options  and 
allows  programme  ‘headroom’)  may  be  offset 
by  situations  in  which  our  adversaries  are 


extremely  adept  at  integrating  and  adapting 
the  best  technologies  from  the  new  global 
market. 

Insurance  vs.  manaairiq  security 

To  summarise,  should  future  defence 
planning  attempt  to  provide  insurance 
against  every  conceivable  risk  -  which  is 
going  to  be  unaffordable  -  or  should  we 
move  to  a  more  sophisticated  security 
planning  strategy  that  seeks  to  manage  risk 
in  a  more  conscious  way? 

Although  some  of  the  serious  risks  that  have 
traditionally  driven  defence  planning  and 
procurement  strategies  still  exist  in  the  post- 
Cold  War  world,  they  are  now  of  a  relatively 
low  probability  and  (in  many  cases)  come 
with  long  warning-times.  Innovations  in  rapid 
procurement  may  be  used  to  mitigate  some 
of  these  risks. 

The  demise  of  Cold  War  bipolarity  and  the 
current  period  of  systemic  transition  have, 
generally  speaking,  presented  a  host  of  more 
minor  risks  with  a  much  higher  probability  of 
occurrence.  It  is  time  to  focus  more 
attentively  on  these,  noting  that  our 
involvement  in  the  potential  crises 
associated  with  them  is  likely  to  be 
discretionary.  Some  form  of  metric  is 
required  to  decide  under  what  conditions  we 
become  involved.  We  will  need  to  tailor 
capabilities  accordingly,  and  deploy  them 
with  agility,  sensitivity  and  precision. 

The  crucial  issues  remain.  What  is  the 
degree  of  overlap?  And,  how  much  should 
we  invest  in  attempting  to  mitigate  each  of 
these  risk  areas?  Possible  implications  for 
the  United  Kingdom  could  be  that  we  invest 
in  a  ‘golf  bag’  of  capabilities,  with  some 
‘clubs’  held  in  quantity  (where  it  is  cost- 
effective  to  do  so)  and  others  kept  latent 
using  advanced  technology  and  research. 
Capabilities  could  be  chosen  on  the  basis  of 
where  the  United  Kingdom  adds  most  value 
to  security,  and  discretionary  activity  could 
be  modulated  by  what  capabilities  we  have 
in  the  ‘golf  bag’. 
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APPENDIX  A 

OUTLINE  OF  MACRO  SCENARIOS 

The  ‘Insight’  macro  scenarios  were 
constructed  using  a  number  of  different 
approaches  that  examined  the  interaction 
between  ‘Political  coherence’  (engaged  or 
disengaged)  and  ‘Alliances  and  alignments’ 
(formal  or  informal)  drivers.  The  third  driver 
dimension  was  ‘Globalisation  of  economic 
systems’  (interdependent  or  fragmented). 
Holding  the  other  five  drivers  constant  as  a 
baseline  assumption,  the  security  aspects  of 
these  four  macro  scenarios  can  be 
summarised  as  follows: 

•  In  the  ‘Web  of  nations’  scenario  there 
are  no  major  regional  blocks  with  limited 
defence  alliances,  implying  that  NATO 
gradually  weakens.  Global  security 
matters  are  increasingly  orchestrated  by 
the  United  Nations  and  enforced  by  ad 
hoc  groups  with  aligned  interests.  The 
effective  marginalisation  of  the  developing 
world  and  underclasses  within  the 
developed  world  lead  to  growing  trouble 
from  sub-national  and  transnational 


groups. 

unlikely. 

Nation-to-nation 

conflict  is 

•  In  the 

‘Subsidiarity’ 

scenario,  the 

realities  of  world  politics  encourage  a 
leaching  of  power  upward  to  regional 
groups  for  international  matters  and 
downward  to  provinces  for  more  local 
matters.  Nations  group  into  effective 
unions  and  global  security  matters  are 
increasingly  dominated  by  a  tripolar 
configuration  of  nation  states  centred  on 
the  American  continent,  Europe  and  Asia. 
The  global  security  oversight  is  similar  to 
the  ‘Web  of  nations’  world,  but  increased 
regional  coherence  deters  outliers  more 
effectively  and  direct  nation  to  nation 
conflict  remains  feasible  though  unlikely. 

•  In  the  ‘Fragmented’  scenario,  the 
process  of  globalisation  and  the  rising 
power  of  international  institutions 
seriously  diminish  the  power  of  the  nation 
state.  While  many  nations  retain  their 
internal  power  and  regional  groups  may 
exist,  the  security  picture  is  dominated  by 
a  plethora  of  non-state  actors  pursuing 
diverse  objectives.  The  probability  of 
inter-nation  conflict  is  higher  here, 
especially  in  troubled  parts  of  the  world, 
and  there  is  a  much  greater  reliance  on 
military  intervention  but  with  a  poor 
risk/reward  balance. 


•  The  ‘Blocs’  scenario  envisages  a  total 
rejection  and  retrenchment  from 
globalisation,  with  powerful  economic  and 
cultural  groups  forming  but  limited 
interaction  between  them.  Governments 
are  likely  to  be  authoritarian,  enjoying 
strong  populist  support  from  their 
electorates  who  favour  extreme  measures 
for  dealing  with  terrorism  and  organised 
crime,  even  at  the  expense  of  individual 
liberties.  The  global  economy  undergoes 
a  period  of  stagnation  and  the  overall 
security  picture  is  characterised  by  a 
stand-off  balance  of  power  between 
factions  that  ameliorates  the  likelihood  of 
crisis  to  some  extent. 
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Scope  of  Operational  Test  and  Evaluation  (OT&E) 

J  Manclark,  HQ  US  Air  Force  Test  &  Evaluation,  USA 
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United  States  Air  Force 
Directorate  of  Test  and  Evaluation 


urpose 


Objectives 

•  Changing  scope  of  operational 
test  and  evaluation  (OT&E) 

Overview 

•  Background 

•  What  has  changed? 

•  Operation  Impact  Assessment 
(OIA)  -  The  new  “E” 

•  Coalition  OT&E 


United  States  Air  Force 
Directorate  of  Test  and  Evaluation 


Baol^qj^und 


The  law 

•  United  States  Code  (USC)  - 
Title  10 

DoD  implementation  of  the  law 
•DoD  Directive  5000.1 
•DoD  Instruction  5000.2-R 
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i)  the  field  test  under  realistic  combat  conditions,  of  any 
item  of  (or  key  component  of)  weapons,  equipment,  or 
munitions  for  the  purpose  of  determining  the 
effectiveness  and  suitability  of  the  weapons,  equipment, 
or  munitions  for  use  in  combat  by  typical  military  users; 
and 

ii)  the  evaluation  of  the  results  of  such  test” 


United  States  Air  Force 
Directorate  of  Test  and  Evaluation 


Implementation  of  the  Law 
DoD5000.2-R>  '  ‘ 


“Operational  test  and  evaluation  (OT&E)  programs  shall  be 
structured  to  determine  . . . 

1 )  the  operational  effectiveness  and  suitability  of  a  system 
under  realistic  conditions  (e.g.,  combat)  and 

2)  if  the  minimum  acceptable  operational  performance 
requirements  as  specified  in  the  ORD  have  been 
satisfied.” 


•■f  ♦  <i' 


United  States  Air  Force 
Directorate  of  Test  and  Evaluation _ 

i  DoD  5000.2-R  ^ 

I  Mandatory  Test  Conditions 

"Threat  or  threat  representative  forces,  targets, 
and  threat  countermeasures,  validated  in 
coordination  with  PI  A,  shall  be  used.” 

"Typical  users  shall  operate  and  maintain  the 
system  or  item  under  conditions  simulating 
combat  stress  and  peacetime  conditions.” 

“The  independent  operational  test  activities  shall 
use  production  or  production  representative 
articles  for  the  dedicated  phase  of  OT&E  that 
supports  the  full-rate  production  decision,  or  for 
ACAT  lA  or  other  acquisition  programs,  the 
deployment  decision.” 


United  States  Air  Force 
Directorate  of  Test  and  Evaluation 


-  DoD5000,2-R  .  •  .  : 

Operational  Assessment  Requirements 


"Whenever  possible,  an  operational  assessment  shall 
draw  upon  test  results  with  the  actual  system,  or 
subsystem,  or  key  components  thereof,  or  with 
operationally  meaningful  surrogates.” 

“When  actual  testing  is  not  possible  to  support  an 
operational  assessment,  such  assessments  may  rely 
upon  computer  modeling,  simulations  (preferably  with 
real  operators  in  the  loop),  or  an  analysis  of  information 
contained  in  key  program  documents." 


2. 


United  States  Air  Force 
Directorate  of  Test  and  Evaluation 


Initial  OT&E  nOT&EV Restrictions 


. . .  initial  operational  test  and  evaluation  shall 
not  comprise  an  operational  assessment 
based  exclusively  on  computer  modeling; 
simulation;  or.  an  analysis  of  system 
requirements,  engineering  proposals,  design 
specifications,  or  any  other  information 
contained  in  program  documents.” 
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•  Introduction  of  Operational  Impact  Assessments 

•  Insights  to  CONORS  limitations,  supporting  systems 
shortfalls  and  joint  system  interoperability 

•  Provided  to  using  commands  and  warfighting  CINCS 

•  Feedback  to  Requirements  Process 

•  Streamlined  acquisition 

•  Reduced  government  testing 

•  “Insight  vice  Oversight” 

•  Simulation  Based  Acquisition 

•  Use  modeling  and  simulation  early  and  throughout  a 
system’s  life 

•  System  and  threat  models  available  for  OT  use 


I  InItPri  Air  Fftrr.fi 

Directorate  of  Test  and  Evaluation 


“The  focus  should  be  on  mission-level  evaluations*, 
assessing  the  impact  of  the  system  under  test  in  its 
operational  environment  and  taking  into  consideration 
all  the  variables  that  can  contribute  to  mission 
success  or  failures.” 


*  Title  10  OT&E  plus  Operational  Impact  Assessment 


s/23/00 


United  States  Air  Force 
Directorate  of  Test  and  Evaluation 


Tests,  Evaluations,  and  Operational 
Impacts  ' 


Title  10 
OT&E 


How  well  does  the 
system  actually  work 
in  expected 
operational 
environments? 


United  States  Air  Force 
Directorate  of  Test  and  Evaluation 


Ooerational  Impact  Assessment: 


Identifies  potential  impacts  on  battlefield  operations 

•  Deployment,  employment  and  sustainment 
considerations 

•  CONOPS/TTP  Issues 

•  Interoperability  with  other  systems  in  the  battlespace 

•  Required  support;  e.g.  communications 

•  Potential  conflicts;  e.g.  interference 

The  question  is:  “Can  the  bridge  be  destroyed  with  the 
given  resources  (weapons,  interfaces,  information, 
etc.)?” 

•  The  question  is  not:  “What  is  the  value  of  destroying 
the  bridge?” 
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United  States  Air  Force 
Directorate  of  Test  and  Evaluation 


System  OT&E  vs  OIA 


United  States  Air  ruiw 
Directorate  of  Test  and  Evaluation 


; Planning  and  Reporting 
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United  States  Air  Force 
Directorate  of  Test  and  Evaluation 


Defining  Coalition  OT&E 


Traditional  OT&E 

•  Does  the  system  meet  expectations  when 
operated  by  typical  personnel  in  the  intended 
environment 

Operational  Impact  Assessment 

•  Are  supporting  systems,  operational  concepts 
and  TTPs  (tactics,  techniques  and  procedures) 
adequate? 

Coalition  OT&E 

•  Can  the  system  perform  in  conjunction  with 
systems  from  other  Services/Nations? 
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United  States  Air  Force 
Directorate  of  Test  and  Evaluation 


and  Coalition  OT&E 


Title  10 
OT&E 


Multi-National  Environment 

Doctrine 

TTPs 

Interoperability 
Electronic  Compatibility 


Coalition  OT&E 


United  States  Air  Force 
Directorate  of  Test  and  Evaluation 


Ghallenqes 


Force  Modernization  and  Interoperability 

•  Force  structure,  doctrine  and  equipment 
differences 

•  Rapidly  changing  technology 

•  Transfer  of  Technology 

•  Lack  of  standardization  for  interoperability  and 
compatibility 
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V 


2.12 


United  States  Air  Force 
Directorate  of  Test  and  Evaluation 


Actions 


Develop  Coalition  OT&E  Concept  for  a  Selected 
Mission  Area 

•  Address  NATO  participants  and  capabilities 

•  Establish  requirements  to  inter-operate  ... 
operational  focus 

Work  toward  more  OT&E  in  Joint  Exercises 

•  Multi-Service 

•  Multi-Nation 

•  In  conjunction  with  Joint  Forces  Command 
Propose  Joint  Test  and  Evaluation  program  as  initial 
step 


United  States  Air  Force 
Directorate  of  Test  and  Evaluation 


Conclusions 


•  OT&E  is  becoming  more  important  ~  and  more  complex 

•  Streamlined  Acquisition 

•  Simulation  Based  Acquisition 

•  The  core  mission  has  not  changed 

•  Evaluate  the  real  system  in  the  real  environment  with 
real  operators  and  maintainers 

•  But,  we  have  expanded  that  mission  to  provide 
assessments  that  are  even  more  operationally  relevant 
to  the  warfighting  CINCs. 

•  Need  to  continue  to  improve  OT&E  to  ensure  relevance 
to  the  acquisition  process  and  to  the  warfighters  by 
implementing  actions  to  meet  the  Commander’s  intent. 

Coalition  warfare  demands  we  test  and  train  together! 
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A  Credible  Guide  to  Future  Warfare 
Lt  Col  Chris  Collett 
Directorate  Force  Development 
Ministry  of  Defence,  UK 


This  presentation  will  eover  three 
areas.  I  will  begin  by  deseribing  how 
the  MOD  is  going  about  establishing 
the  “credible  guide”  that  the  title 
claims  in  order  to  aid  its  medium  and 
long  term  plarming.  I  will  then  make  a 
most  un-MOD  like  leap  by  describing 
firstly  some  of  the  characteristics  of 
future  warfare  and  then  secondly 
highlight  some  of  the  key  factors  in 
those  conflicts.  These  last  two  sections 
are  my  personal  views  based  on  work 
carried  out  over  the  last  two  years  and 
do  not  represent  any  official  MOD 
position. 

EVOLUTION  OF  WARFARE 
(EOWl  PROJECT 

The  EOW  project  originated  some 
three  years  ago  when  28  academic 
papers  were  produced  looking  at 
sources  of  instability  out  to  2015. 
Those  papers  formed  much  of  the 
intellectual  basis  for  the  subsequent 
Strategic  Defence  Review  and  were  the 


starting  point  for  the  current  project. 
EOW’s  aim  is  as  follows: 

"To  provide  a  credible  view  of  future 
warfare  and  other  military  operations 
in  order  to  better  inform  MOD  long 
term  decision  making.  ” 

EOW  attempts  to  look  out  thirty  years 
and  therefore,  has  to  consider  an 
immense  range  of  possibilities  but  it  is 
not  just  an  interesting  exercise  it  is 
intended  to  influence  real  life 
decisions.  EOW  must,  therefore, 
simultaneously  look  widely  whilst 
applying  a  rigorous  “so  what”  analysis 
to  the  concepts  that  emerge  and 
thereby  provide  a  robust  audit  trail  for 
decisions.  Hence  the  cyclical  structure 
shown  below  wherein  one  year’s 
general  strands  of  analysis  produces 
direction  for  the  next  year’s  specific 
studies  the  outcome  of  which  is  fed 
into  the  following  year’s  general 
studies. 


EOW 

Strands  of  Analysis 

Year  YearB  Year 

Genei;al  <jGenei;al  ^General 

\lnforms  \jnJ6rms 

/ UlKects  / DiJ^ts 

Specific  Specific  Specific 
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The  general  strands  of  analysis 
include. 

Academic  papers  commissioned 
from  experts  outside  the  MOD. 

Internal  papers  produced  by  MOD 
or  other  government  departments 

Scenario  decomposition/analysis 
wherein  teams  of  military, 
scientific  and  other  experts  dissect 
specially  developed  scenarios  in 
order  to  highlight  what  tasks  would 
need  to  be  carried  out  and  what 
broad  order  capabilities  would  be 
needed  to  accomplish  them. 

Wider  technology  work  which 
exploits  the  future  technology 
studies  being  carried  out  by  DERA. 

Outreach  by  which  is  meant  that 
we  seek  to  capture  the  views  of  as 
many  others  in  the  field  as 
possible. 

The  specific  strands  of  analysis  can 
include: 

Technology  studies  carried  out  in 
partnership  with  the  DERA’s 
Technical  Top  Team 

Other  Specially  Commissioned 
Studies. 

Conflict  simulation  games  that 
allow  us  to  explore  in  more  depth 
novel  technologies,  strategies  or 
tactics. 

Collaborative  Projects. 


SOME  CHARACTERISTICS  OF 
FUTURE  CONFLICTS 

Here  I  step  beyond  reporting  what  the 
MOD  is  doing  to  give  you  some  of  my 
own  personal  (i.e.  unendorsed)  views 
about  the  future  based  on  the  work  we 
have  carried  out  over  the  last  two 
years. 

Firstly,  what  sort  of  conflicts  are  we 
likely  to  see?  I  would  suggest  that  we 
are  unlikely  to  be  involved  in 
traditional  force  on  force  conflicts 
along  the  lines  of  a  strategic  attack  on 
NATO  or  close  ally  because  of  the 
conventional  superiority  we  (by  which 
I  mean  the  Atlantic  Alliance)  now 
enjoy  courtesy  largely  of  the  US.  This 
is  likely  to  endure  but  with  perhaps 
narrowing  margins  due  to  a  shrinking 
force  structure  less  able  to  deal  with 
multiple  concurrent  demands.  Were 
this  superiority  to  be  lost,  things  might 
be  different  and  the  opportunistic 
competitor  who  takes  advantage  of 
either  a  temporary  technical  edge  or 
our  distraction  remains  a  possibility. 

What  are  far  more  likely  are 
interventions  of  choice  where  national 
survival  is  not  at  stake  and  even  the 
national  interests  are  open  to  debate. 
My  colleague  Dr  Hamid,  has  gone  into 
detail  on  why  we  are  likely  to  see 
plenty  of  potential  for  strife  in  the 
future  and  beyond  this  brief  reminder  I 
do  not  propose  elaborate  and  certainly 
not  to  disagree  with  his  conclusions.  I 
would  venture  to  say  that  this  broad 
analysis  is  shared  by  at  least  the  major 
western  countries. 
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Sources  of  Strife 

Population  Pressure,  Urbanisation, 
Poverty 
Resources 

Alienation  along  political,  economic, 

ethnic,  religious  and  “virtual”  lines 

Empowerment  of  non  state  actors 

It  might  be  argued  that  we  will 
exercise  our  choice  and  not  get 
involved  with  what  are  likely  to  be 
confusing,  intractable  and  dangerous 
situations.  I  suggest  however,  that 
there  are  a  number  of  reasons  why  on 
many  occasions  it  will  not  be  possible 
to  ignore  conflicts  or  other  crises. 

Firstly,  the  humanitarian  impetus  is 
likely  to  remain  and  indeed  grow  as 
global  communications  continue  to 
expand.  However,  how  fragile  this 
impetus  will  be  or  indeed,  whether  a 
general  “fatigue”  will  set  in  is 
debatable. 

Globalisation  will  continue  and 
probably  increase  the  impact  of  local 
instability  on  the  world  economy  and 
UK  citizens’  wellbeing.  Today,  10 
million  UK  passport  holders  reside 
abroad  and  40  million  travel  overseas 
each  year  and  this  seems  likely  to 
grow. 

Environmental  damage  is  not  confined 
to  borders  and  as  industrialisation 
spreads  the  potential  for  accidental  or 
deliberate  damage  arising  from 
conflict. 

Global  communications,  travel  and 
general  migration  means  that  conflicts 
can  be  expected  to  spill  over  into  other 
nations  even  more  so  than  we  have 
seen  to  date.  The  mass  movement  of 
refugees  is  of  course,  better  avoided 
than  prevented  or  dealt  with  once 
underway. 


It  is  important  to  also  recognise  why 
we  may  not  be  able  to  intervene. 
Domestic  support  is  naturally  crucial 
but  an  increasing  degree  of 
international  agreement  is  also  likely  to 
be  needed  particularly  as  so  many 
forces  are  likely  to  have  multinational 
commitments.  Interventions  of  choice 
imply  a  cost/benefit  analysis  must  be 
carried  out  and  it  is  likely  that  in  the 
future  this  will  be  available  far  more 
reliably,  far  more  in  advance  of  an 
operation  and  to  far  more  people.  How 
soon  before  the  evening  news’ 
computer  graphics  depict  not  only  the 
military  options  open  to  allied  forces 
but  their  cost  in  terms  of  lives  and  cash 
perhaps  with  the  latter  able  to  be 
represented  in  units  of  the  viewers 
choice,  so  many  hospitals  for  example? 
Finally,  there  will  be  many  demands 
and  limited  resources.  Choosing  to 
commit  to  one  operation  will  have  to 
be  done  with  an  appreciation  of  its 
impact  on  the  ability  to  commit  to  the 
next  one. 

Having  painted  a  picture  of  a  world 
with  plenty  demand  for  military  action 
most  of  which  will  involve  a  finely 
balanced  choice  I  will  now  suggest 
some  of  the  options  open  to  those  who 
would  oppose  such  interventions. 

Initially  an  opponent  will  probably  try 
to  prevent  an  intervention  ever 
beginning  by  perhaps,  raising  the 
perception  of  the  likely  cost  and  by 
attacking  the  base  of  support  domestic 
and  international.  If  that  does  not 
seem  likely  to  work,  he  would  be 
better  to  deny  the  intervening  force 
physical  access  to  the  theatre  either 
through  political  pressure  on  host 
nations,  covert  disruption  of  lines  of 
communications  or  overt  military 
action  or  the  threat  of  it.  Once  faced 
with  a  deployed  force  an  opponent  is 
likely  to  attack  the  consensus  that 
keeps  it  there  perhaps  by  concentrating 
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strikes  on  one  relatively  weak  member 
of  a  coalition.  Finally,  he  may  aim  for 
sheer  physical  survival  with  the 
potential  to  conduct  further  offensive 
operations  and  hope  to  outlast  the 
intervention  force.  The  audience  will 
be  well  able  to  think  of  weapons  and 
tactics  that  could  be  used  to  prosecute 
such  a  layered  strategy  many  of  them 
falling  under  the  rather  catch-all  term 
of  asymmetric  warfare. 

Having  now  painted  in  another  piece 
of  the  picture  (whether  it  is  credible  or 
not  is  for  the  audience  to  decide)  I 
would  now  like  to  suggest  some  of  the 
factors  that  will  be  key  in  future 
operations. 

INFORMATION  OPERATIONS 

Achieving  and  maintaining 
information  superiority  or  indeed 
dominance  will  be  of  critical 
importance.  It  is  the  key  enabler 
needed  to  achieve  rapid  success  in 
complex  environments  (physical  and 
political)  with  less  risk  and  fewer 
unwanted  consequences.  To  achieve 
this  capability  will  require  the  fusion 
of  multiple  sources  of  information 
including  both  multispectral  real-time 
sensors  and  diverse  databases,  to  give 
commanders  and  planners  a  level  of 
knowledge  such  that  they  are  confident 
of  predicting  the  actions  of  parties  in 
an  operation  (friendly,  enemy  and 
neutral). 

Operators  at  all  levels  also  need  to  be 
able  to  accurately  predict  the  outcome 
of  their  own  actions.  This  capability 
needs  to  encompass  everything  from 
the  physical  results  of  a  single 
weapon’s  arrival  to  its  impact  on  the 
opinions  of  all  significant  populations. 
Achieving  such  a  capability  is 
therefore,  far  more  than  simply 
developing  ever  more  capable  sensors 
and  suggests  that  more  resources  will 


need  to  be  devoted  to  “soft”  data 
collection,  storage,  expert  retrieval  and 
comprehensive  adaptive  modelling 
systems.  The  alternative  to  “operating” 
smarter  is  to  operate  (including  to 
fight)  with  a  heavier  hand  than  may  be 
financially  or  politically  acceptable. 

A  major  concern,  however,  is  that  the 
level  of  operational  expertise  called  for 
above  may  be  equally  available  to  any 
opponent  able  to  exploit  commercial 
technology.  It  will  be  increasingly 
important  to  track  the  availability  of 
real  time  information  and  expert 
systems  in  order  to  assess  the  threat  to 
our  operations  and  to  develop 
countermeasures.  The  importance  of 
information  and  communications  in  the 
civil  world,  technical  progress  and  the 
degree  of  redundancy  becoming 
available  may  make  denial,  disruption 
or  physical  deception  impossible  in 
which  case  more  subtle  techniques 
may  be  required.  Clearly,  as 
opponents  inevitably  acquire  longer 
range  and  more  destructive  weapons  it 
becomes  more  important  to  control 
their  ability  to  use  them  if  we  are  to 
retain  freedom  of  action  with 
acceptable  risk.  The  alternatives 
would  be  to  either  increase  the 
physical  protection  of  more  of  our 
forces  or  place  ever  greater  effort  in 
pre-emptive  destruction  of  threat 
systems.  These  two  strategies  are 
likely  to  be  expensive  and/or 
politically  unacceptable. 

MINIMAL  DEPLOYED  FORCES 

Minimising  the  physical  presence  of 
deployed  forces  reduces  the 
transportation  and  sustenance  bills 
while  obviously  reducing  the  number 
of  persoimel  at  most  risk.  It  is 
therefore,  a  vital  goal  to  which 
increased  use  of  stand-off  or 
uninhabited  systems,  robots  and  higher 
levels  of  equipment  reliability  will  all 
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contribute.  However,  the  other  side  of 
the  coin  is  that  the  remaining  forces 
that  still  need  to  go  in  harm’s  way  (and 
the  size  of  this  area  is  likely  to  expand) 
will  be  more  fragile  thus  increasing  the 
need  for  information  dominance  and 
other  means  of  force  protection.  Those 
forces  that  must  operate  within  reach 
of  enemy  systems  (particularly  in 
Peace  Support  Operations  where 
physical  presence  is  still  expected  to  be 
critical)  will  also  have  to  have  assured 
access  to  support  from  afar  that  can 
match  their  timeliness  demands.  There 
are  technological  developments  that 
could  provide  this  but  they  are  not 
likely  to  remove  the  requirement  for  on 
board  sensors,  weapons  and  defences 
completely. 

STRATEGIC/TACTICAL 

MOBILITY 

Both  strategic  and  tactical 
transportation  systems  need  to  be  much 
faster  to  both  improve  the  chances  of 
achieving  surprise  and  to  minimise  the 
time  their  elements  spend  in  areas  of 
high  risk.  This  desire  for  speed 
includes  loading,  unloading  and  turn 
around  times.  Developing  systems  that 
are  less  tied  to  traditional  infrastructure 
such  as  ports,  airstrips  or  railheads  will 
be  particularly  important.  Such 
facilities  can  be  considered  to  be 
increasingly  vulnerable  in  the  future 
and  being  tied  to  them  increases  the 
difficulty  of  achieving  surprise. 

Ideally  forces  (and  their  logistic 
support)  should  be  delivered  from 
strategic  distances  almost  directly  to 
their  objectives,  which  places  an 
increased  onus  on  protecting  whatever 
transport  systems  used  and  blurs  the 
distinction  between  strategic  and 
tactical  assets.  Indeed  if  deployed 
forces  could  have  organic  theatre  range 
mobility  it  would  offer  major 
operational  advantages. 


TAILORED  PRECISION  STRIKE 
CAPABILITY 

The  trend  towards  enhanced  precision 
being  required  from  weapon  systems  is 
not  going  to  diminish.  Striking  must 
also  be  timely  and  restricted  to  the 
target  and  effect  required.  Inducing 
shock  will  be  as  equally  desirable  as 
achieving  physical  disablement  as  it 
adds  to  force  protection  and  increases 
the  speed  of  conflict  termination. 
Conversely,  the  ability  to  “safe” 
weapons  after  release  will  be  important 
and  the  ability  to  record  the  decision 
making  process  that  led  to  their  use 
(even  down  to  individual  rifle  shots) 
will  be  critical  when  inevitably  called 
upon  to  justify  actions. 

The  advantages  of  using  non  lethal 
weapons  against  equipment  are  clear 
particularly  when  it  was  necessary  to 
neutralise  an  area  to  be  passed  through 
or  to  disrupt  opponent’s  C3  system. 
The  ability  to  use  less  than  lethal  force 
against  personnel  to  achieve  temporary 
freedom  of  action  will  also  be  needed 
but  is  much  harder  to  achieve  in  an 
acceptable  manner.  The  advantages  of 
using  non  lethal  weapons  are  highly 
dependent  on  the  previously  discussed 
ability  to  accurately  and  confidently 
predict  their  results. 

FORCE  PROTECTION 

Information  dominance  is  the  most 
critical  element  of  force  protection  as 
long  as  it  is  coupled  with  the  ability  to 
either  remove  a  threat  capability  before 
it  is  employed  or  to  avoid  its  effects. 
Negating  or  living  with  the  effects  of 
future  attacks  will  be  increasingly 
difficult  as  lethality  rises  and 
acceptance  of  losses  declines. 


3.5 


Chemical  and  biological  hazards  will 
continue  to  be  major  concerns.  It  is 
likely  that  the  UK  itself  will  be 
threatened  by  painful  even  if  not 
militarily  significant  attacks  (e.g.  cyber 
or  terror  attacks)  as  opponents  seek  to 
prevent  or  punish  interventions.  The 
nuclear  deterrent  may  not  be  credible 
against  what  might  be  relatively  low 
levels  of  threatened  destruction  and 
therefore  other  forms  of  tailored  and 
scaleable  deterrents  will  be  needed 
(e.g.  financial). 
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REORGANIZING  AND  REALIGNING  DEFENSE  ORGANIZATIONS 

Ernest  A  Seglie 
Science  Advisor, 

Operational  Test  and  Evaluation 
Office  of  the  Secretary  of  Defense 
Washington,  D.C,  USA 


Abstract 

Many  countries  are  in  the  process  of 
implementing  acquisition  reform.  The  United 
States  has  changed  many  aspects  of  its 
acquisition  process  including  the  organization 
of  its  testing  organizations,  both  at  the  Office 
of  the  Secretary  of  Defense  level  and  in  the 
Services. 

This  paper  reviews  these  changes  in  the 
testing  organizations  and  processes. 

The  Evolution  of  Defense  Organizations 

Napoleon  is  reputed  to  have  said,  "When  all 
else  fails,  reorganize."  It  is  ironic  that  the 
Western  powers  have  all  begun  huge  efforts  to 
reorganize  since  the  end  of  the  Cold  War.  The 
Defense  Systems  Management  College  in  the 
United  States,  as  part  of  the  International 
Defense  Educational  Agreement  (IDEA)  forum, 
studied  the  acquisition  systems  in  the  UK, 
Germany,  France  and  the  United  States.  Their 
first  conclusion  was  that  all  the  countries  have 
experienced  or  are  currently  implementing  some 
level  of  acquisition  process  reform.  Why? 

The  reason  they  ascribe  is  the  perception  that 
it  takes  too  long  and  costs  too  much  to  acquire 
new  weapons  systems. 

Within  the  United  States  the  evidence  in 
favor  of  such  a  perception  is  supplied  if  we  look 
at  the  cost  and  schedule  overruns  for  programs. 
We  asked  the  DSMC  to  investigate  the  details  in 
those  acquisition  programs  for  which  the 
Director  of  Operational  Test  and  Evaluation  had 
to  provide  a  report  to  Congress.  These  are  major 
programs,  for  which  the  total  program  cost  is 
over  a  Billion  Dollars,  often  tens  of  Billions. 

We  can  display  the  relative  overruns  by 
plotting  on  one  axis  the  ratio  of  actual  cost  to 
complete  the  phase  to  the  planned  cost.  On  the 
other  axis  is  the  ratio  of  the  actual  time  to 
complete  the  phase  to  the  planned  time.  This 
study  is  of  the  final  phase  of  acquisition,  from 
the  Milestone  II  decision  (at  which 
Demonstration,  Validation,  and  Risk  Reduction 


is  over  and  Engineering  and  Manufacturing 
Development  is  authorized)  to  the  decision  to  go 
to  full-rate  production,  called  Milestone  III.  If 
the  program  manager  had  succeeded  in 
delivering  what  was  promised  at  cost  and  on 
time,  each  ratio  would  be  one,  and  the  program 
would  be  represented  as  a  program  at  the  point 
(1,1).  The  actual  plot  for  the  programs  is: 
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For  those  major  acquisition  systems  that 
produced  DOT&E  reports  to  Congress,  the 
average  cost  over  run  is  40%  and  the  average 
schedule  over  run  is  62%. 

Some  comments  are  in  order. 

For  those  of  you  who  were  raised  on  the 
management  dictum  that  time  is  money,  you 
might  expect  that  the  overruns  should  fall  on  the 
diagonal  with  cost  overrun  about  the  same  as 
schedule  overrun  on  a  per  cent  basis.  This 
appears  not  to  be  the  case,  but  the  deviation  can 
be  explained.  For  some  programs,  in  particular 
those  that  gave  evidence  of  problems  early;  the 
Congress  will  sometimes  impose  a  cost  cap. 
This  puts  an  upper  bound  on  cost.  Difficulties  in 
the  program  then  appear  as  slowing  the  spend 
rate.  What  often  happens  in  this  case  is  that  the 
program  itself  changes.  For  example  the  B-1 
was  produced  within  the  cost  cap  but  without  a 
defensive  electronics  suite.  This  type  of  trade¬ 
off  now  goes  by  the  rubric  "Cost  as  an 
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Independent  Variable"  in  the  Department  of 
Defense. 

So  this  is  the  presenting  condition  for  which 
a  solution  is  to  be  found.  Note  that  there  might 
be  two  ways  to  solve  this  apparent  problem. 
First  is  to  get  a  better  estimate  at  Milestone  II  of 
what  the  cost  and  schedule  would  actually  be. 
The  second  is  to  try  to  contain  the  cost  and 
schedule  to  meet  the  promised  cost  and  schedule 
identified  at  Milestone  II.  Most  of  the 
reorganization  within  the  U.S.  Department  of 
Defense  has  focused  on  the  latter.  We  should 
note  that  most  studies  of  the  matter,  a  Defense 
Science  Board  Study  in  the  late  1980's  and  a 
RAND  Study,  have  indicated  that  the  reason  for 
the  overrun  is  a  very  optimistic  estimate  of  the 
technical  difficulty  of  the  project. 

One  can  imagine  the  pressures  on  programs 
to  be  optimistic  it  all  the  estimates.  There  is 
competition  among  the  bidders  when  the 
programs  request  proposals  for  contractors. 
There  is  competition  among  programs  for 
continued  funding  even  after  the  program 
begins. 

These  are  not  pressures  that  would  be 
unexpected  in  other  areas  of  human  endeavour 


where  the  project  is  large.  In  fact  performance 
of  such  large  programs  has  been  analyzed. 

This  study  of  pre-1985  projects  has  about  the 
same  cost  overrun  as  the  post  1983  programs  in 
the  acquisition  study. 

The  Different  Organizations  in  US  Defense 
The  organizations  that  had  a  stake  in  testing 
and  Defense  in  the  U.S.  include: 

•  the  Congress, 

•  the  Director,  Operational  Test  and 
Evaluation, 

•  The  Undersecretary  of  Defense  for 
Acquisition,  Technology  and  Logistics  (and 
at  the  time  within  that  organization  a 
Director  for  Research  and  Engineering  and 
under  the  DDR&E  a  Director  for  Test, 
Systems  Engineering,  and  Evaluation.  One 
of  the  major  changes  was  the 
disestablishment  of  the  later  office.)  In  the 
mid-80s,  the  Congress  established  a  second 
T&E  office  within  the  Office  of  the 
Secretary  of  Defense  that  provided  a 
different  kind  of  oversight  of  the 
Army/Navy/ Air  Force  Services  from  the  first 
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office.  It  was  established  at  the  Assistance 
Secretary  level  to  provide  an  independent 
operational  T&E  assessment  prior  to 
initiating  production  of  any  weapon  system. 

•  The  Services  (Army,  Navy,  Marines  and  Air 
Force). 

Within  each  Service  there  is  an  acquisition 
element  and  a  testing  element.  Each  of  these  has 
undergone  some  change.  Probably  the  Army 
has  done  the  most  reorganizing.  (The  Army 
reorganization  was  the  model  for  the 
recommendations  for  change  made  by  the 
Defense  Science  Board.) 

The  Game-Theoretic  Approach  to 
Information  (A  National  Academy  of  Sciences 
review  of  Test  and  Evaluation! 

The  evolution  of  institutions  in  DoD 
Operational  Testing  was  studied  in  a  game- 
theoretic  model  by  members  of  a  National 
Academies  of  Science  panel  that  reviewed 
testing  and  acquisition  for  the  Department.  The 
model  is  of  persuasion  games  in  which  an 
interested  party  (the  agent)  possess  information 
regarding  the  state  of  nature  and  attempts  to 
influence  a  decision-maker  by  selectively 
providing  data. 

Even  under  the  assumption  that  the  agent 
never  makes  an  incorrect  report,  the  agent  may 
make  reports  that  range  from  entirely 
uninformative  to  absolutely  accurate.  Early 
work  showed  that  an  equilibrium  exists  in  which 
the  decision  maker  resolves  to  ignore  all  reports 
and  the  agent  makes  only  uninformative  reports. 

The  model  was  extended  to  consider  the  case 
in  which  the  agent  can  choose  whether  to 
become  informed.  When  disclosure  is 
mandatory,  the  agent  may  choose  not  to  acquire 
information,  and  to  claim  ignorance  (or  use 
some  other  basis  for  claims). 

The  next  stage  of  the  game  is  for  the 
decision-maker  to  take  a  stance  of  extreme 
scepticism  on  any  claim  of  ignorance  by  the 
agent.  The  agent  can  counter  if  the  cost  of 
acquiring  information  is  a  privately-held 
information  by  the  agent.  Then  claims  about  the 
cost  of  acquiring  information  can  give 
credibility  to  the  claims  of  ignorance  by  the 
agents.  The  notion  of  cost  is  here  a  general  one 
and  you  can  consider  cost  to  include  time.  Thus 
the  claim  that  it  would  take  too  long  to  acquire 


information  is  equivalent  to  the  claim  of 
excessive  cost  to  acquire  information. 

Secretary  of  Defense  Initiatives  with  respect  to 
Test  and  Evaluation 

As  recommended  by  both  Secretary  Cohen 
and  Secretary  Perry,  early  involvement  by 
operational  testers  provides  early  feedback  to 
help  acquisition  programs  address  operational 
issues.  As  the  Undersecretary  of  Defense  for 
Acquisition  Technology  and  Logistics 
(USD(A,T&L))  explained  in  a  letter  to  the 
chairmen  of  the  four  Defense  Committees,  "I 
have  advocated  for  many  years  that  serious 
testing  with  a  view  toward  operations  should  be 
started  early  in  the  life  of  a  program.  Early 
testing  against  operational  requirements  will 
provide  earlier  indications  of  military 
usefulness.  It  is  also  much  less  expensive  to 
correct  flaws  in  system  design,  both  hardware 
and  software,  if  they  are  identified  early  in  a 
program.  Performance-based  acquisition 
programs  reflect  our  emphasis  on  satisfying 
operational  requirements  vice  system 
specifications." 

To  implement  these  policies,  the  Secretary  of 
Defense  decided  to  disestablish  the  office  of 
Director,  Test,  Systems  Engineering  and 
Evaluation  (D,TSE&E)  within  the  Office  of  the 
Under  Secretary  of  Defense  for  Acquisition  and 
Technology,  and  move  the  following 
responsibilities  to  DOT&E: 

•  Central  Test  and  Evaluation  Investment 
Program. 

•  Joint  Technical  Coordinating  Group  for 
Munitions  Effectiveness. 

•  Joint  Technical  Coordinating  Group  on 
Aircraft  Survivability. 

•  Threat  Systems  Office. 

•  Precision  Guided  Weapons  Countermeasure 
Test  Directorate. 

•  Test  and  Evaluation  Independent  Activities. 

The  Secretary  further  directed  that  DOT&E: 
(1)  exercise  no  responsibility  with  respect  to 
Developmental  Test  and  Evaluation  of  a  defense 
acquisition  program,  except  to  advise  those  who 
do  have  such  responsibility;  (2)  ensure  that  all 
contractor  support  is  consistent  with  limitations 
of  10  U.S.C.  2399;  (3)  ensure  that  all  actions 
taken  are  without  regard  to  whether  the 
particular  range  or  equipment  is  used  for  DT&E 
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or  OT&E;  and  (4)  ensure  that  no  action  is  taken 
that  creates  a  conflict,  potential  conflict,  or 
appearance  of  conflict  of  interest  with  respect  to 
DOT&E's  role  which  is  independent  of  DT&E. 

Oversight  of  developmental  tests  remains  in 
USD(A,T  &L).  Joint  Test  and  Evaluation 
(JT&E)  also  remains  in  E1SD(A,T&L).  This 
activity  was  considered  a  special  case  because, 
while  JT&E  projects  are  not  acquisition  projects 
in  and  of  themselves,  they  should  help  identify 
needed  acquisition  programs.  However,  the 
JT&E  program  was  specifically  chartered  to 
provide  a  mechanism  for  joint  operational  tests 
that  support  the  Commanders-in-Chiefs  (CINCs) 
and  the  Services  in  developing  more  effective 
ways  of  employing  fielded  systems  in  a  joint 
operational  environment.  To  retain  an 
operational  test  focus,  the  Secretary  has 
determined  that  JT&E  projects  will  be  jointly 
chartered  by  the  Director,  Strategic  and  Tactical 
Systems  (D,S&TS)  within  USD(A,T&L)  and 
DOT&E.  Also,  as  in  the  past,  DOT&E  will 
approve  all  JT&E  program  test  plans  jointly  with 
D,S&TS.  The  Joint  T&Es  are  reviewed  in  the 
main  body  of  this  report. 

The  Secretary  also  directed  that  DoD 
Directive  3200.11,  "Major  Range  and  Test 
Facilities  Base  (MRTFB),"  be  revised 
appropriately  to  reflect  the  realignment  of 
responsibilities  to  reflect  that  DOT&E  will 
establish  policy  for  and  composition  of  the 
MRTFB,  and  plan,  program,  and  budget  for  the 
Central  Test  and  Evaluation  Investment 
Program. 

In  addition,  the  DOT&E  charter  (DoD 
Directive  5142.1)  is  being  updated  to  reflect 
these  changes. 

After  reorganization  last  year  we  hope  for 
numerous  benefits  to  the  Department  of 
Defense’s  T&E  oversight  role  such  as: 

•  A  single  voice  to  the  Congress  on  matters  of 
T&E  policy,  oversight  and  budget  defense 

•  A  single  voice  of  oversight  to  the  military 
departments  of  the  Army,  Navy  and  Air 
Force 

•  A  single  point  of  focus  for  test  range 
modernization  and  T&E  technology 
leadership 

A  Defense  Science  Board  study  is  underway 
that  will  recommend  to  the  Congress  what  more 
needs  to  be  done  to  improve  the  efficiency  and 
productivity  of  the  T&E  infrastructure 


operations.  This  study  is  scheduled  to  report  out 
to  the  Congress  in  August,  but  it  is  likely  that 
their  recommendations  will  be  in  line  with  those 
reported  out  last  year  by  the  first  DSB  study 
group,  such  as: 

•  Start  T&E  planning  early — ^very  early 

•  Make  T&E  part  of  the  acquisition  process — 
not  adversarial  to  it 

•  Consolidate  DT  and  OT 

•  Provide  joint  test  leadership 

•  Fund  modeling  and  simulation  support  of 
T&E  in  program  budgets 

•  Maintain  independence  of  evaluation 
process  while  integrating  all  other  activities 

•  Establish  range  ownership  and  operation 
structure  separate  from  the  Service  DT/OT 
organization 

Conclusion 

For  those  of  you  who  must  reorganize 
and  wish  it  to  be  successful,  I  would  suggest  that 
you  hire  a  game  theorist  and  social 
anthropologist.  The  Social  anthropologist  will 
work  out  the  reward  system  for  players  in  the 
game.  Not  as  it  is  supposed  to  be,  but  as  it  really 
is.  And  the  game  theorist  should  be  able  to  tell 
you  how  the  rules  and  reorganization  will  be 
“played”  within  that  reward  system. 
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Abstract 

This  paper  discusses  the  United  States 
Air  Force  perspective  concerning  the  role  that 
Test  and  Evaluation  (T&E)  plays  in  the  United 
States  (U.S.)  defense  acquisition  process.  For 
the  U.S.  Air  Force,  T&E  is  clearly  an  integral 
part  of  the  acquisition  process,  and  is  involved  in 
every  phase  from  concept  exploration  to 
production.  By  supplying  timely,  accurate,  and 
affordable  information,  the  T&E  community 
provides  the  Program  Manager  with  the 
information  needed  to  make  decisions  critical  to 
the  success  of  the  program. 

This  paper  also  emphasizes  how  Air 
Force  Materiel  Command  (AFMC)  conducts 
developmental  test  and  evaluation  (DT&E), 
which  along  with  operational  test  and  evaluation 
(OT&E),  are  the  two  principal  types  of  T&E 
employed  by  the  U.S.  Air  Force.  Key  to  the 
manner  in  which  AFMC  conducts  DT&E  is  the 
constant  effort  to  operate  in  a  more  cost-effective 
and  time-efficient  manner,  while  never  forgetting 
that  it  is  a  military  organization  whose  primary 
mission  is  to  support  the  warfighter. 

Introduction 

Good  afternoon.  Considering  the 
ability  of  this  forum  to  draw  so  much  talent  from 
the  international  T&E  community,  I  consider  it  a 
great  privilege  to  be  here  speaking  to  you  today. 

I  would  like  to  thank  the  Royal  Aeronautical 
Society  and  the  International  Test  and  Evaluation 
Association  (ITEA)  for  organizing  this  forum 
and  for  inviting  me  here. 

Before  I  begin  my  formal  remarks,  I 
would  first  like  to  start  with  a  short  video  clip 
that  highlights  the  immense  aerospace  power  that 
the  U.S.  Air  Force  brings  to  the  battlefield. 

Many  elements  are  necessary  to  build  an  airforce 
of  this  magnitude,  and  I  am  especially  proud  of 
the  critical  role  that  the  U.S.  Air  Force  T&E 
community  has  had  in  making  possible  the  awe¬ 
inspiring  weapon  systems  you  will  see  in  this 
video. 

The  world  of  U.S.  Air  Force  T&E  is 
fascinating  and  dynamic,  and  at  times  even 
frustrating,  but  in  the  brief  time  I  have  to  speak,  I 
want  to  give  you  some  insight  on  the  U.S.  Air 
Force’s  philosophy  concerning  T&E,  and  the 


manner  in  which  we  use  it  to  help  build  some  of 
the  finest  weapon  systems  in  the  world. 
Specifically,  I  hope  to  be  able  to  convey  to  you 
this  morning  that  for  the  U.S.  Air  Force,  T&E  is 
an  integral  part  of  the  acquisition  process.  The 
timely,  accurate,  and  affordable  information 
provided  by  the  T&E  community  allows  the 
Program  Manager,  also  known  as  the  Single 
Manager,  to  make  critical  lifecycle  decisions.  In 
addition,  I  want  to  emphasize  that  in  today's 
environment  of  acquisition  reform,  which  rightly 
seeks  to  acquire  weapon  systems  "better, 
cheaper,  and  faster,"  the  need  for  T&E  is  just  as 
critical  as  ever.  In  fact,  as  many  program 
managers  can  tell  you,  there  is  probably  no 
"better,  cheaper,  and  faster"  way  to  jeopardize 
your  program  than  to  neglect  the  requirement  for 
adequate  testing. 

Overall  Role  of  U.S.  Air  Force  T&E 

When  discussing  the  role  that  U.S.  Air 
Force  T&E  plays  in  the  procurement  of  weapon 
systems,  it  is  crucial  to  point  out  that  in  the  U.S. 
defense  establishment,  we  make  a  significant 
distinction  between  DT&E  and  OT&E.  How 
does  DT&E  differ  from  OT&E,  and  what  role  do 
they  play  in  the  procurement  process?  Quite 
simply,  the  U.S.  Air  Force  uses  DT&E  to  learn 
and  confirm,  that  is,  to  learn  about  the  system's 
capabilities,  and  to  confirm  that  the  system 
performs  according  to  the  specifications.  With 
this  knowledge,  the  Single  Manager  can  evaluate 
the  program  for  risk  and  determine  the  technical 
challenges  that  must  be  addressed  to  keep  the 
program  on  track.  Since  AFMC  is  responsible 
for  developing  and  acquiring  the  U.S.  Air  Force's 
weapon  systems,  it  is  AFMC  that  conducts 
DT&E. 

Qn  the  other  hand,  the  U.S.  Air  Force 
uses  OT&E  to  answer  two  fundamental 
questions.  First,  given  a  realistic  environment, 
can  the  warfighter  use  the  system  to  accomplish 
the  mission?  Second,  given  the  same  realistic 
environment,  can  the  warfighter  support  and 
maintain  the  system?  In  the  language  of  U.S. 

Air  Force  T&E,  the  first  question  measures 
operational  effectiveness,  while  the  second 
measures  operational  suitability.  The  clear 
distinction  we  make  between  DT&E  and  OT&E 
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recognizes  the  fact  that  while  a  weapon  system 
may  meet  all  the  design  specifications,  it  may 
still  fail  to  accomplish  the  mission.  Just  as 
important,  U.S.  law  dictates  that  the  agency 
responsible  for  OT&E  must  be  independent  of 
the  command  that  develops  the  weapon  system. 
Thus,  while  AFMC  is  responsible  for  DT&E,  the 
Air  Force  Operational  Test  and  Evaluation 
Center  (AFOTEC)  carries  the  OT&E  mission  for 
the  U.S.  Air  Force. 

T&E  is  Integral  to  the  Acquisition  Process 
One  of  the  main  points  I  want  to 
emphasize  is  that  for  the  U.S.  Air  Force,  T&E 
plays  an  integral  role  in  the  acquisition  process. 
Perhaps  Dr  Jacques  S.  Gansler,  Under  Secretary 
of  Defense  for  Acquisition  and  Technology, 
captured  the  U.S.  Air  Force  T&E  perspective 
best  when  he  addressed  the  ITEA  Symposium 
last  year.  In  his  address  Dr  Gansler  stated 

"We  must  produce  our  weapon 
systems  on  greatly  reduced  cycles 
. . .  with  greatly  enhanced 
performance  ...  at  greatly  reduced 
cost.  Testing  throughout  the 
development  process  is  a  key  to  our 
success  in  these  objectives.  This 
puts  our  testing  community  ...  as 
an  integral  part  of  the  acquisition 
team." 

The  whole  idea  is  that  for  the  U.S.  Air 
Force,  T&E  isn't  something  that  comes  at  the  end 
of  the  acquisition  process,  that  is,  it  isn't  a  type  of 
pass/fail  exam  or  a  process  of  inspecting  a 
finished  product  for  quality.  Rather,  the  T&E 
community  should  be  involved  from  the  very 
beginning  of  the  acquisition  cycle,  and  the 
timely,  accurate,  and  affordable  information 
T&E  provides  to  decision  makers  is  critical  to 
the  success  of  the  program. 

DT&E  Role  in  Acquisition  Process. 

Let's  first  discuss  the  three  primary  roles  that 
DT&E  plays  in  the  acquisition  process,  and  why 
these  roles  are  so  valuable  to  the  Single  Manager 
and  other  decision  makers.  First,  DT&E  acts  as 
the  essential  feedback  loop  within  the  systems 
engineering  process.  Second,  it  provides  the 
Single  Manager  with  an  independent  perspective 
of  the  system.  And  finally,  DT&E  serves  as  a 
bridge  between  the  technical  community  that 
designs  the  system,  and  the  operational 
community,  which  must  use  that  system. 

Within  the  U.S.  acquisition  community, 
we  make  extensive  use  of  the  systems 


engineering  process  (Figure  1),  which  is  an 
iterative  problem-solving  process  that  translates 
the  warfighter's  needs  into  performance 
thresholds  and  objectives.  Thresholds  represent 
the  minimum  capabilities  that  the  weapon  system 
must  achieve,  while  objectives  represent  the 
desired  capabilities.  These  thresholds  and 
objectives  are  then  used  to  determine  functional 
and  performance  requirements,  which  then 
translate  into  design  solutions,  and  eventually 
end  up  as  drawings  and  system  specs.  This 
process  is  repeated  constantly  as  the  design 
system  evolves  from  models  to  components,  and 
then  to  complete  systems. 

What  is  DT&E's  role  in  the  systems 
engineering  process?  Clearly,  DT&E  acts  as  the 
essential  feedback  loop.  In  fact,  DT&E  is  the 
only  way  to  insure  that  the  design  meets  the 
warfighter’s  operational  needs  as  originally 
identified  in  the  thresholds  and  objectives.  If  a 
program  does  not  have  adequate  DT&E,  the 


Figure  1 

Systems  Engineering  Process 


Single  Manager  will  simply  not  be  able  to  verify 
that  the  system  meets  the  warfighter's 
requirements.  To  the  extent  that  the  Single 
Manager  cannot  verify  that  the  system  meets  the 
warfighter's  requirements,  he  will  incur  a 
corresponding  increase  in  risk  associated  with 
fielding  the  system.  The  system  may  not 
perform  as  designed  and  could  even  end  up 
having  disastrous  results.  Therefore,  the  Single 
Manager  should  clearly  recognize  the  value  of 
DT&E  as  a  primary  tool  for  risk  reduction.  In 
addition,  the  valuable  data  that  DT&E  provides 
to  the  Single  Manager  will  allow  him  or  her  to 
have  a  rationale,  informed  basis  for  making 
tradeoffs  between  cost  and  performance. 

Secondly,  DT&E,  and  specifically  U.S. 
Air  Force  DT&E,  as  opposed  to  contractor 
DT&E,  provides  the  Single  Manager  with  an 
objective  and  independent  assessment  of  system 
performance.  This  independent  assessment  is 
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absolutely  critical  to  the  success  of  any  program. 
In  this  sense,  the  U.S.  Air  Force  DT&E 
community  acts  as  the  Single  Manager's  trusted 
agent,  providing  him  or  her  with  independent 
reporting  on  system  performance  and  spec 
compliance,  and  insuring  that  the  so  called  “bad 
news”  will  not  get  buried  or  suppressed.  This 
degree  of  independence  and  objectivity  also 
gives  the  U.S.  Air  Force  DT&E  community  a 
great  degree  of  credibility  with  Operational  Test 
Agencies  (OTA),  such  as  AFOTEC.  Thus,  when 
the  DT&E  community  certifies  that  a  system  is 
ready  for  OT&E,  the  OT&E  community  knows  it 
can  tmst  their  assessment.  This  credibility  also 
means  that  the  OT&E  community  can  tmst  a 
DT&E  assessment  to  focus  operational  testing  on 
specific  problem  areas. 

Lastly,  U.S.  Air  Force  military  DT&E 
testers  typically  possess  operational  experience 
and  therefore  are  able  to  speak  the  language  of 
both  the  engineer  and  the  operator.  In  this  sense, 
the  U.S.  Air  Force  DT&E  test  community  has  a 
unique  understanding  of  both  the  design 
constraints  and  the  user’s  requirements.  Thus, 
the  Single  Manager  has  a  unique  and  vital 
resource  that  he  can  and  should  use  to  insure  the 
success  of  his  program. 

OT&E  Role  in  Acquisition  Process. 
Moving  to  the  role  that  OT&E  plays  in  the 
acquisition  process,  it  is  important  to 
reemphasize  that  the  U.S.  Air  Force  involves  the 
OT&E  community  from  the  very  beginning  of 
the  acquisition  process  (Figure  2).  Early  in  the 
program,  the  OT&E  community  helps  formulate 
useful  and  well-defined  weapon  system  program 
requirements.  Prior  to  Milestone  (MS)  II,  that 
portion  of  the  acquisition  cycle  that  essentially 
deals  with  concept  exploration  and  preliminary 
design,  the  U.S.  Air  Force  operational  test 
community  will  conduct  Early  Operational 
Assessments  (EOA)  to  forecast  and  evaluate  the 
potential  operational  effectiveness  and  suitability 
of  the  weapon  system  during  development. 

After  MS  II,  in  which  the  program  moves  into  a 
detailed  design  phase,  the  operational  test 
community  conducts  Operational  Assessments 
(OA)  to  evaluate  the  system's  potential  to  meet 
mission  requirements,  and  to  help  support  a 
decision  to  begin  initial  production,  known 
formally  as  a  low  rate  initial  production  (LRIP) 
decision.  With  the  production  of  the  first  few 
production  articles,  the  operational  testers  begin 
Initial  OT&E  (lOT&E),  which  is  conducted  on 
the  LRIP  articles  to  insure  that  the  program  is 
ready  to  enter  full-rate  production.  After  MS  111, 
or  the  full-rate  production  phase,  the  operational 


test  community  conducts  follow-on  OT&E 
(FOT&E)  to  confirm  corrections  of  deficiencies 
noted  in  lOT&E,  and  to  evaluate  major 
modifications  to  the  system. 


Figure  2 

T &E  and  the  Acquisition  Process 
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How  AFMC  conducts  DT&E. 

It  is  now  time  to  turn  our  attention  to 
the  manner  in  which  AFMC  actually  carries  out 
DT&E  for  the  U.S.  Air  Force.  Some  of  the 
concepts  I  will  discuss,  such  as  Single  Face  To 
the  Customer  (SFTC)  and  Responsible  Test 
Organization  (RTO)  are  unique  to  the  way  the 
U.S.  Air  Force  conducts  DT&E.  Other  concepts, 
such  as  the  Combined  Test  Force  (CTF)  and 
Combined  Developmental  Test/Operational  Test 
(DT/OT),  apply  to  all  U.S.  military  services.  Of 
course,  in  these  days  of  reduced  budgets  and 
shrinking  forces,  AFMC  is  constantly  seeking 
new  ways  to  support  the  warfighter  in  a  more 
cost-effective  and  time-efficient  manner.  But 
regardless  of  how  we  conduct  DT&E  for  the 
U.S.  Air  Force,  I  want  to  stress  that  AFMC  is 
first  and  foremost  a  military  organization.  While 
we  strive  to  insure  our  decisions  make  sense 
from  a  business  perspective,  there  are  times 
when  the  "commercial  or  business  solution"  will 
simply  not  apply.  The  need  to  accomplish  the 
mission  will  always  override  the  desire  to 
implement  practices  that  make  sense  from  a 
business  perspective  only. 

Single  Face  To  the  Customer  (SFTCj 
Concept.  We  all  know  that  the  world  of  T&E  is 
a  specialized  discipline,  and  although  it  is  clearly 
critical  to  the  success  of  any  acquisition 
program,  the  typical  Single  Manager  does  not 
have  the  in-house  expertise  to  take  care  of  all  his 
or  her  T&E  planning  needs.  For  that  very 
reason,  the  U.S.  Air  Force  has  established  the 
SFTC  office,  which  serves  as  the  first  stop  for  all 
Single  Managers,  providing  comprehensive,  top- 
level  T&E  planning  from  a  non-geographic  or 
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test  center  perspective.  1  should  point  out  that 
the  SFTCs  also  serve  as  the  initial  point  of 
contact  for  U.S.  contractors  as  well.  We  have 
five  SFTC  offices,  which  correspond  to  the  five 
test  mission  areas:  Airframe-Propulsion- 
Avionics  (APA),  Armament/Munitions  (A/M), 
Command  and  Control  (C2),  Electronic  Warfare 
(EW),  and  Space.  In  a  very  real  sense,  the 
SFTCs  are  an  extension  of  my  headquarters  staff, 
and  are  physically  located  at  the  AFMC  test 
centers  where  the  predominance  of  T&E 
resources  related  to  that  specific  mission  area 
resides. 

Each  of  the  SFTC  offices  are  staffed 
with  knowledgeable  and  experienced  personnel 
who  stay  current  on  the  latest  U.S.  Air  Force 
T&E  instructions,  directives,  processes,  and 
trends.  In  addition,  the  staff  also  has  access  to 
operations,  range,  logistics,  and  program 
personnel  to  provide  a  full  spectrum  of  technical 
and  program  support.  From  a  mission  area 
perspective,  the  SFTCs  are  able  to  provide  the 
Single  Manager  with  information  on 
measurement  facilities  (MF),  hardware-in-the- 
loop  (HITL)  facilities,  system  integration 
laboratories  (SIL),  installed  system  test  facilities 
(ISTF),  and  open  air  ranges  (OAR)  throughout 
the  Department  of  Defense  (DoD).  And  each 
SFTC  is  well  equipped  to  identify  and 
recommend  the  optimum  combination  of  test 
resources  that  are  available  throughout  the  U.S. 
Air  Force  and  DoD.  Whether  developing  a  new 
system,  or  working  a  major  modification, 
upgrade,  or  product  improvement  program,  the 
Single  Manager  can  tap  the  appropriate  SFTC  for 
initial  T&E  planning. 

Responsible  Test  Organization  (RTO) 
Concept.  Once  the  initial  T&E  planning  is 
underway,  one  of  the  most  important  duties  of 
the  SFTC  is  to  provide  a  prioritized  list  of 
recommended  Responsible  Test  Organizations, 
or  RTOs,  to  the  Single  Manager.  What  exactly 
is  an  RTO?  Essentially,  an  RTO  is  the 
government  agency  responsible  for  the  oversight 
of  the  planning,  safety,  conduct,  and  reporting  of 
developmental  test  programs.  Notice  that  I  said 
government  agency,  and  that  "conducting  tests" 
is  not  the  exclusive  right  of  the  RTO.  Many 
times  only  the  contractor  has  the  expertise  to 
actually  conduct  the  test.  But  in  keeping  with 
the  philosophy  that  DT&E  is  an  inherently 
government  responsibility,  only  a  government 
agency  can  be  the  RTO,  not  the  contractor. 
Typically,  RTOs  are  test  organizations  within 
U.S.  Air  Force  test  centers.  Air  Logistics 
Centers,  or  other  service  test  organizations. 


During  the  development  of  a  system,  a  program 
may  need  several  RTOs  for  different  phases  of 
the  program,  but  there  will  only  be  one  RTO  for 
a  specific  phase  at  any  given  time.  As  the 
Director  of  Operations  for  AFMC,  only  I  have 
the  authority  to  formally  designate  a  specific 
organization  as  the  RTO,  though  I  make  this 
decision  in  collaboration  with  the  Single 
Manager  and  the  commanders  of  the  test 
organizations.  Once  I  designate  the  RTO,  the 
SFTC  transfers  all  DT&E  planning  responsibility 
to  the  RTO. 

Combined  Test  Force  Concept  ICTF). 
As  the  agency  responsible  for  the  detailed-level 
planning  of  the  Single  Manager's  test  program, 
the  RTO  is  fully  expected  to  use  the  CTF 
concept  to  the  greatest  extent  possible.  The 
CTF,  which  is  comprised  of  contractor,  DT&E, 
OT&E,  and  operating  command  personnel,  can 
be  located  at  one  primary  test  site  or  distributed 
across  multiple  test  sites,  and  seeks  to  get  the 
biggest  T&E  "bang"  for  the  increasingly  limited 
"buck"  available  for  testing.  By  sharing  assets, 
analysts,  support  staff,  and  test  events,  the 
seemingly  separate  test  objectives  of  the 
contractor  and  U.S.  Air  Force  testers,  both 
DT&E  and  OT&E,  can  be  combined  to  realize 
savings  in  both  time  and  money.  Combined 
DT/OT  is  perhaps  the  most  obvious  example  of 
how  the  CTFs  save  time  and  money.  An 
example  of  combined  DT/OT  could  be  a 
prototype  free-fall  munition  that  is  released  from 
a  fighter  aircraft  at  operational  employment 
conditions  instead  of  from  a  static  stand,  thus 
satisfying  both  DT  and  OT  objectives. 

The  advantages  of  the  CTF  are  many 
and  include  a  decrease  in  the  time  required  for 
testing,  and  therefore  a  shortened  acquisition  life 
cycle,  elimination  of  redundant  activities  and  a 
corresponding  savings  in  cost,  and  early 
involvement  of  OT&E  personnel  during  system 
development.  Early  involvement  of  OT&E 
personnel  increases  their  familiarity  with  the 
system  and  permits  them  to  convey  operational 
concerns  to  the  developer  in  time  to  allow 
changes  in  system  design. 

Before  leaving  this  topic,  I  want  to 
emphasize  that  the  CTF,  and  especially  the  push 
for  combined  DT/OT,  is  not  a  mixing  and 
matching  of  DT/OT  objectives.  The  CTF  fully 
realizes  the  importance  of  recognizing  separate 
DT  and  OT  objectives  where  necessary.  Clearly, 
the  requirement  for  the  DT&E  and  OT&E 
communities  to  maintain  a  degree  of 
independence  resides  not  in  collecting  separate 
test  data,  but  in  the  independent  analysis  and 
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reporting  of  that  test  data.  While  not  forgetting 
that  there  must  always  be  a  phase  of  dedicated 
OT&E,  the  CTF  concept  allows  the  operational 
testers  to  get  involved  early  where  they  can  have 
an  important  impact  in  highlighting  concerns  at  a 
time  when  they  can  be  more  easily  corrected. 

T&E  and  Commercial-Off-The-Shelf 
(COTSt/Non-Developmental  Items  (NDiy  In 
many  ways,  the  U.S.  defense  community  has 
greatly  benefited  from  COTS/NDI.  Through 
COTS/NDI,  we  have  greatly  reduced  the  time  to 
field  a  system,  saved  scarce  R&D  funds,  and 
incorporated  state-of-the-art  technology  much 
sooner.  On  the  other  hand,  COTS/NDI  presents 
certain  challenges  as  well,  especially  from  a 
T&E  perspective.  Typically,  COTS/NDI 
acquisitions  are  used  in  an  environment  different 
from  the  one  for  which  they  were  originally 
designed.  Secondly,  engineering  and  test  data  is 
often  not  available.  Therefore,  as  the  acquisition 
community  turns  to  COTS/NDI  because  of  its 
genuine  advantages,  it  is  important  that  the  T&E 
community  help  the  Single  Manager  understand 
the  need  for  adequate  testing.  More  importantly, 
perhaps,  the  T&E  community  itself  must 
formulate  a  strategy  for  better  integrating 
COTS/NDI  into  the  T&E  process.  I  think  we 
can  all  agree  that  COTS/NDI  is  an  area  that  has 
presented  challenges  that  we  in  the  T&E 
community  have  not  been  able  to  answer 
completely,  but  one  that  we  cannot  ignore. 

International  Test  and  Evaluation 
Programs.  Given  the  international  audience  here 
today,  I  think  it  especially  appropriate  to  discuss 
the  immense  benefits  that  all  U.S.  military 
services  have  reaped  from  the  international  T&E 
programs  we  have  with  our  allies.  For  example, 
the  Foreign  Comparative  Test  (FCT)  program 
allows  us  to  evaluate  a  foreign  nation's  weapon 
system  or  technology  in  terms  of  its  potential  to 
meet  a  requirement  for  our  own  military 
services.  As  U.S.  testers  work  with  the  test 
community  from  other  nations  to  evaluate  the 
existing  test  data  and  to  validate  test  data 
collected  during  our  evaluation,  the  U.S  military 
services  not  only  save  T&E  funds,  but  also 
procure  weapon  systems  already  developed  and 
in  use  by  our  allies.  To  eite  only  a  few 
examples,  under  the  North  Atlantic  Treaty 
Organization  (NATO)  Comparative  Test 
Program,  the  U.S.  has  procured  the  Norwegian 
Penguin  missile,  a  United  Kingdom-designed 
minehunter,  and  a  German  mine  reconnaissance 
and  detection  system. 

Going  beyond  FCTs  in  terms  of 
international  T&E  cooperation,  the  U.S.  testing 


community  has  also  has  worked  closely  with  our 
NATO  allies  on  several  cooperative  programs. 
For  example,  during  the  U.S.  and  NATO 
Airborne  Warning  and  Control  System 
(AWACS)  Electronic  Support  Measures  (ESM) 
program,  which  greatly  improved  enemy 
identification  and  combat  surveillance 
capabilities,  both  U.S.  and  NATO  E-3As  were 
used  as  test  aircraft  in  a  combined  DT/OT 
program.  A  Joint  Test  Force  (JTF),  comprised  of 
members  from  the  U.S.  and  NATO  test 
community,  conducted  testing  in  both  the  U.S. 
and  European  theaters.  In  an  example  that 
directly  impacted  our  recent  conflict  in  Kosovo, 
testers  from  the  U.S.,  Belgium,  Denmark,  the 
Netherlands,  and  Norway  worked  together  and  at 
an  accelerated  schedule  to  upgrade  the  European 
Participating  Air  Forces'  (EPAF)  F-l  6s.  Taking 
the  EPAF  F-l 6s  from  1970s  to  1990s 
technology,  the  upgrade  allowed  the  aircraft  to 
operate  with  LANTIRN  targeting  pods  and  laser- 
guided  bombs,  thus  providing  our  European 
allies  with  increased  combat  capabilities  in  direct 
support  of  vital  NATO  missions  during 
Operation  ALLIED  FORCE.  In  today's  world  of 
coalition  warfare,  where  weapons 
standardization  and  interoperability  within 
NATO  are  increasingly  important,  these 
cooperative  programs  are  critical. 

Further  proof  of  the  advances  the  U.S. 
and  its  NATO  allies  have  made  in  the  area  of 
international  T&E  cooperation  can  be  seen  by 
the  numerous  agreements  the  U.S.  now  has  with 
France,  Germany,  and  the  United  Kingdom  to 
accept  each  country's  T&E  data,  and  thereby 
avoid  redundant  testing.  Along  these  same  lines, 
cooperative  efforts  by  France,  Germany,  the 
United  Kingdom,  and  the  U.S.  have  resulted  in 
International  Test  Operations  Procedures 
(ITOPs),  documents  containing  mutually 
acceptable  standardized  test  procedures.  The 
ITOPs  include  state-of-the-art  test  procedures, 
and  their  use  ensures  high  quality,  efficient, 
accurate,  and  cost  effective  testing.  The  latest 
effort  by  the  ITOPs  community  has  centered  on 
standardizing  Modeling  and  Simulation  (M&S) 
test  procedures,  an  area  that  has  become 
increasingly  important  to  all  testers  worldwide. 
The  success  of  ITOPs  can  be  seen  by  the  fact 
that  non-signatory  allies  such  as  Israel,  Sweden, 
and  Spain  routinely  use  ITOPs  in  developing 
their  military  systems.  In  summary,  the  ITOPs 
program  is  yet  another  excellent  example  of  how 
the  U.S.  and  NATO  test  communities  are 
working  together  to  shorten  acquisition 
timelines,  minimize  duplicate  testing,  improve 
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interoperability  of  U.S.  and  allied  equipment, 
and  exchange  advanced  test  technology. 

Funding  DT&E.  I  think  it  is  important 
to  finish  my  discussion  on  the  manner  in  which 
AFMC  conducts  DT&E  by  discussing  how 
AFMC  actually  funds  DT&E.  U.S.  Air  Force 
DT&E  is  funded  by  a  combination  of  customer 
(or  direct)  and  institutional  (or  indirect)  funding 
sources.  Customers  pay  the  direct  costs  such  as 
direct  labor,  flying  hours,  test  facility  costs,  and 
a  proportional  share  of  certain  other  costs  (like 
range  use  costs).  We  use  institutional  sources  to 
pay  for  our  military  personnel,  center  staffs,  base 
support,  indirect  test  costs,  and  to  sustain  our  test 
infrastructure.  This  method  of  funding  seems  to 
strike  the  right  balance-affordable  testing  for 
our  customers  while  sustaining  our  extensive  test 
infrastructure.  This  balance  between 
institutional  and  customer  sources  is  common  to 
all  members  (Army,  Navy,  and  Air  Force)  that 
operate  under  the  Major  Range  and  Test  Facility 
Base  designation. 

With  acquisition  reform,  we  see  more 
DoD  contractors  making  the  important  decisions 
of  whether  or  not  to  test,  and  where  to  test. 

These  decisions  greatly  impact  our  DT&E 
capabilities.  We  have  certain  test  capabilities 
that  are  national  assets,  assets  that  we  must 
maintain  regardless  of  whether  or  not  there  is  a 
near-term  demand  for  the  capability.  Of  course, 
it's  best  if  we  have  customers  paying  our  testers' 
salaries.  However,  if  the  contractors  decide  to 
test  somewhere  else,  or  not  test  at  all,  we  still 
have  to  maintain  a  cadre  of  experienced  testers. 
In  these  circumstances,  our  goal  is  to  keep  the 
minimum  number  of  testers  needed  to  maintain 
the  capability.  In  recent  years  we  have  sought  to 
reduce  the  cost  of  maintaining  this  cadre  by 
cross-training  our  testers  so  that  they  are  multi- 
skilled  and  able  to  support  more  than  one  test 
capability.  We  have  reduced  the  number  of  test 
support  aircraft  and  constantly  address  the  size 
and  capability  of  the  infrastructure  we  are  tasked 
to  maintain.  The  bottom  line  is  we  are 
continuously  looking  to  balance  the  need  to 
maintain  critical  test  capabilities  with  the  desire 
to  operate  in  a  cost-efficient  manner. 

Summary 

Let  me  conclude  by  reemphasizing 
some  of  the  key  themes  I  have  discussed  today. 
First,  it  should  be  clear  that  T&E  plays  an 
integral  role  in  the  acquisition  process.  T&E 
provides  the  Single  Manager  with  the  critical 
information  he  needs  to  make  the  key  decisions 
necessary  for  a  successful  program.  And 


secondly,  through  our  SFTCs,  RTOs,  and  CTF 
concept,  AFMC  strives  to  conduct  T&E  in  the 
most  effective  and  efficient  manner  possible 
given  our  limited  resources.  However,  even  as 
we  do  our  best  to  make  smart  business  decisions 
in  implementing  T&E,  we  will  never  let  our 
desire  to  run  an  efficient  T&E  business  override 
our  need  to  support  the  warfighter.  Thank  you. 
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Glossary 


AFMC  -  Air  Force  Materiel  Command 

AFOTEC  -  Air  Force  Operational  Test  and  Evaluation  Center 

A/M  -  Armament/Munitions 

APA  -  Airframe-Propulsion-Avionics 

AWACS  -  Airborne  Warning  and  Control  System 

C2  -  Command  and  Control 
CTF  -  Combined  Test  Force 

DoD  -  Department  of  Defense 

DT&E  -  Developmental  Test  and  Evaluation 

DT/OT-  Developmental  Test/Operational  Test 

EOA  -  Early  Operational  Assessment 
EPAF  -  European  Participating  Air  Forces 
ESM  -  Electronic  Support  Measures 
EW  -  Electronic  Warfare 

FCT  -  Foreign  Comparative  Test 

FOT&E  -  Follow-On  Operational  Test  and  Evaluation 

HITE  -  Hardware-In-The-Loop 

lOT&E  -  Initial  Operational  Test  and  Evaluation 
ISTF  -  Installed  System  Test  Facility 
ITEA  -  International  Test  and  Evaluation  Association 
ITOPs  -  International  Test  Operations  Procedures 

JTF  -  Joint  Test  Force 

LRIP  -  Low  Rate  Initial  Production 

MF  -  Measurement  Facility 
MS  -  Milestone 

M&S  -  Modeling  and  Simulation 

NATO  -  North  Atlantic  Treaty  Organization 

OA  -  Operational  Assessment 
OTA  -  Operational  Test  Agency 
OT&E  -  Operational  Test  and  Evaluation 

R&D  -  Research  and  Development 
RTO  -  Responsible  Test  Organization 

SETC  -  Single  Face  To  the  Customer 
SIL  -  System  Integration  Laboratory 

T&E  -  Test  and  Evaluation 

U.S.  -  United  States 
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Abstract 

The  Army  Research  Laboratory  (ARL)  has  imple¬ 
mented  an  innovative  program  for  partnering  with 
the  private  sector  called  the  Federated  Laboratory 
(FedLab).  Driven  by  the  requirement  to  develop 
technology  in  areas  where  the  private  sector  has 
both  the  technical  lead  and  a  substantial  investment, 
ARL  decided  to  use  a  relatively  new  instrument,  the 
Cooperative  Agreement,  to  fund  consortia  of  indus¬ 
try  and  universities  to  work  in  partnership  with 
government  researchers.  This  innovative  program 
has  been  under  way  for  four  years.  Strong  collabora¬ 
tive  relationships  have  been  established  with  these 
consortia  of  industry  and  university  partners,  and 
they  are  pursuing  high-quality  science  and  technol¬ 
ogy.  The  collaborative  nature  of  these  partnerships 
has  greatly  facilitated  not  only  the  transfer  of  knowl¬ 
edge  within  FedLab,  but  also  the  transfer  of  innova¬ 
tive  technologies  to  Army  and  other  customer 
applications.  The  program  has  been  recognized  by 
the  Office  of  the  Vice  President  with  a  Hammer 
Award  as  an  example  of  reinventing  the  way  the 
government  does  business.  The  Army  and  ARL 
intend  to  build  on  these  programs  and  initiate  a  new 
expanded  FedLab  that  will  produce  key  technologies 
to  enable  the  new  Army  Vision. 

Introduction 

As  the  Army’s  corporate  laboratory  and  a  member  of 
the  Army  Materiel  Command  (AMC)  team,  the  U.S. 
Army  Research  Laboratory  (ARL)  provides  materiel 
readiness  through  innovative  technology.  Its  mission 
is  to  provide  the  Army  with  the  key  technologies  and 
analytical  support  that  are  necessary  to  ensure 
supremacy  across  the  spectrum  of  military  opera¬ 
tions.  ARL  conducts  basic  and  applied  research 
through  a  variety  of  arrangements  with  external 
organizations,  including  industry,  academia,  foreign 
laboratories,  U.S.  National  Laboratories,  and  other 
Service  laboratories.  ARL  also  conducts  in-house 
research  and  analysis  with  a  staff  of  approximately 
1250  scientists  and  engineers.  ARL’s  primary 
customers  are  the  AMC  Research,  Development, 
and  Engineering  Centers  (RDECs),  Army  Program 
Managers,  and  the  Army  Battle  Labs. 

ARL  has  a  long  history  of  establishing  partnerships 
with  external  organizations  to  seek  the  best  solu¬ 
tions  and  technologies  for  Army  needs.  Typically, 
these  partnerships  are  accomplished  via  contracts, 
grants,  cooperative  research  and  development 
agreements  (CRADAs),  and  memoranda  of  agree¬ 


ment.  Over  the  past  four  years,  ARL  has  developed  a 
unique  series  of  public-private  partnerships  through 
the  use  of  Cooperative  Agreements  with  consortia  of 
industry  and  university  partners.  These  partnerships 
have  formed  the  Army’s  Federated  Laboratory 
(FedLab)  initiative. 

Why  FedLab? 

In  the  mid  1 990’s,  it  became  apparent  that  for  the 
Army  to  maintain  dominance  on  the  future  battlefield, 
information  technology  would  be  essential.  Moreover, 
it  was  also  clear  that  much  of  this  technology  was 
developing  at  an  extremely  rapid  pace,  primarily 
driven  by  private  sector  investments  in  research. 
Although  this  development  is  essential  to  Army 
needs,  it  was  generally  not  directly  applicable,  since 
the  Army  often  requires  technology  to  operate  in 
conditions  and  environments  different  from  the 
typical  commercial  environment  (usually  more 
severe).  For  example,  consider  cellular  communica¬ 
tion  on  the  battlefield.  Whereas  commercial  sector 
technology  is  based  on  a  fixed  (and  unstressed) 
infrastructure,  an  Army  application  would  require 
networks  and  infrastructure  that  are  highly  mobile, 
that  can  be  reconstituted  on  a  nearly  continuous 
basis  as  nodes  move  in  and  out  of  the  network,  and 
that  can  survive  in  an  active  battlefield  environment. 
As  a  result,  additional  research  beyond  ongoing 
private  sector  research  was  required  to  fully  address 
Army  requirements. 

The  challenge  for  the  Army  was  to  develop  a  mecha¬ 
nism  whereby  Army-unique  research  could  be 
coupled  with  ongoing  private  sector  research  so  that 
the  Army  could  take  full  advantage  of  commercial 
technologies  at  the  same  rapid  pace  that  they  were 
being  applied  commercially.  It  clearly  did  not  make 
sense  to  attempt  to  duplicate  the  private  sector’s 
capability  in-house.  However,  in-house  research  that 
focused  on  Army-unique  problems  was  essential  to 
ensure  proper  coupling  of  the  Army  and  private 
sector  research  and  to  transition  commercial  re¬ 
search  to  Army  applications. 

The  FedLab  program  was  devised  by  the  Army  and 
ARL  to  address  this  critical  need.  Through  a  rela¬ 
tively  new  instrument  known  as  a  Cooperative 
Agreement  (not  a  CRADA),  ARL  funded  public- 
private  partnerships  with  universities  and  academia. 
The  Cooperative  Agreement  allowed  ARL  to  form 
true  partnerships  with  the  private  sector  to  jointly 
plan  and  execute  research  and  (most  importantly)  to 
transition  the  technology  developed  to  Army  applica- 
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tions.  (In  contrast,  more  traditional  contractual  or 
grant  arrangements  require  an  “arm’s  length”  rela¬ 
tionship  with  the  vendor.)  The  FedLab  program 
capitalizes  on  the  strong  lead  the  commercial  sector 
has  in  information  technology,  augmenting  it  with 
innovative  research  from  the  academic  sector  and 
focused  research  from  government  laboratories,  to 
accomplish  military-specific  objectives. 

What  is  FedLab? 

FedLab  has  transformed  ARL  into  a  geographically 
distributed  virtual  laboratory  consisting  of  consortia 
of  industry  and  university  partners  augmenting  the 
core  in-house  research  program.  Under  ARL  leader¬ 
ship,  the  private  sector  research  program  is  tightly 
integrated  with  ARL  in-house  research  to  focus  on 
Army  requirements  and  applications.  Each  consor¬ 
tium  consists  of  industry  and  university  members 
with  an  industry  lead.  Scientists  and  engineers  from 
industry  partners  provide  their  knowledge  of  ongoing 
commercial  product  research  and  development,  as 
well  as  a  focus  on  technology  transition  opportuni¬ 
ties.  University  members  bring  their  commitment  to 
cutting-edge  research  and  an  education  focus,  which 
provides  a  constant  stream  of  fresh  innovative 
thinking  in  the  form  of  graduate  students  and  their 
professors.  The  industry  partners  harvest  the 
research  efforts  of  the  universities  for  insertion  into 
products.  Besides  contributing  their  own  unique 
research  skills,  ARL  researchers  keep  the  program 
oriented  towards  Army-specific  problems. 

After  an  extensive  competition,  ARL  signed  Coop¬ 
erative  Agreements  with  three  consortia  and  pro¬ 
vided  funds  for  industry  and  academic  partners  to 
conduct  research  with  ARL  in  support  of  Army 
needs.  The  use  of  Cooperative  Agreements  provides 
ARL  the  flexibility  to  react  to  newly  discovered 
research  opportunities,  as  well  as  to  evolving  Army 
needs  for  technology.  Cooperative  Agreements  also 
allow  ARL  researchers  to  work  closely  with  private 
sector  partners  in  planning  and  executing  joint 
research.  Virtually  every  research  project  or  task 
within  the  FedLab  program  is  jointly  planned  and 
executed  by  a  Principal  Investigator  from  ARL  and 
one  from  at  least  one  industry  and  university  partner. 
The  Cooperative  Agreement  allows  a  fully  integrated 
management  approach  to  program  planning  and 
execution  in  FedLab.  it  also  acts  as  a  five-year 
roadmap  for  the  program:  Research  goals  and 
objectives  are  developed  jointly  by  ARL  and  the 
consortium  and  documented  in  the  Cooperative 
Agreement. 

Each  Consortium  is  managed  by  a  Consortium 
Management  Committee  (CMC),  which  is  chaired  by 
the  industry  lead  and  consists  of  representatives 


from  each  private  sector  partner.  All  consortium 
activities  are  managed  through  the  CMC,  which 
typically  meets  quarterly.  The  CMC  also  approves 
yearly  research  plans  for  submission  to  and  final 
approval  by  the  ARL  Cooperative  Agreement 
Manager  (CAM),  who  participates  in  the  CMC  in  an 
ex-officio  capacity.  These  Annual  Program  Plans 
(APPs)  document  detailed  research  goals  for  the 
year.  The  CAM  ensures  that  the  consortium  research 
is  properly  integrated  with  the  overall  ARL  research 
program.  This  integrated  approach  to  program 
planning  and  execution  ensures  that  research 
performed  by  the  private  sector  partners  augments 
and  effectively  extends  that  of  the  core  ARL  in-house 
program. 

Several  innovative  approaches  have  been  adopted 
in  FedLab  to  encourage  and  enhance  collaboration 
among  the  private  sector  partners  and  government 
in-house  researchers.  These  include  provisions  for  a 
collaborative  environment  using  web-based  tech¬ 
nologies  and  video  teleconferencing,  extensive  staff 
rotations  of  ARL  and  consortia  researchers,  and 
educational  initiatives. 

The  consortia  have  established  secure  web-based 
environments  for  collaborative  research.  This  allows 
consortium  and  government  researchers  to  post  and 
share  research  results,  including  ongoing  research, 
published  papers  and  reports,  annual  research 
plans,  calendars  of  events,  discussion  groups,  and 
interactive  files  and  data.  For  example,  multiple 
researchers  from  several  different  organizations 
collaborate  in  the  development  of  advanced  image- 
processing  algorithms  by  sharing  databases  and 
algorithm  modules  over  the  web.  In  addition,  exten¬ 
sive  use  of  video  teleconferencing  has  enabled 
researchers  to  conduct  informal  meetings,  seminars, 
and  lectures  remotely  as  a  means  of  sharing  infor¬ 
mation  and  enhancing  collaboration. 

Both  long-  and  short-term  staff  rotations  have  been 
used  to  enhance  collaboration  among  research 
partners  and  ARL.  Over  40  researchers  have  been 
on  long-term  rotation,  with  many  more  on  short-term 
assignments.  This  has  greatly  facilitated  the  flow  of 
ideas  and  research  results.  In  many  cases,  opportu¬ 
nities  for  collaboration  were  discovered  only  after 
researchers  spent  time  working  in  other  laboratories. 
For  example,  a  collaboration  combining  acoustic 
sensor  research  at  ARL,  self-power  generation 
research  at  MIT,  and  signal  processing  at  Lockheed 
Martin  resulted  in  a  novel  capability  to  monitor  the 
soldier’s  physiological  condition  without  the  need  for 
batteries.  This  opportunity  resulted  from  a  short-term 
rotation  of  MIT  researchers  at  ARL,  during  which 
time  they  discovered  research  at  ARL  complemen¬ 
tary  to  ongoing  research  at  MIT. This  and  other 
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similar  cases  have  resulted  in  many  successfui 
transfers  of  technoiogy  among  partners  and  to  Army 
and  other  external  customers. 

The  FedLab  partnerships  have  provided  a  significant 
source  of  educational  opportunities  for  ARL  re¬ 
searchers.  Consortium  members  have  established  a 
variety  of  formal  and  informal  education  programs 
including  feliowship  programs  leading  to  advanced 
degrees,  short  courses  in  critical  technology  areas 
relevant  to  the  FedLab  research  areas,  and  lectures, 
seminars,  and  \workshops  in  FedLab  research 
topics.  In  addition,  a  symposium  is  held  annually  at 
which  ARL  and  consortium  researchers  present 
research  accomplishments  over  the  past  year  via 
presentations,  posters,  and  technology  demonstra¬ 
tions.  These  annuai  symposia  also  provide  an 
opportunity  to  show  how  the  work  of  the  three 
consortia  is  being  integrated  towards  the  overall 
FedLab  goal  of  fostering  the  digitization  of  the 
battlefieid. 

Current  Programs 

The  Cooperative  Agreements  for  the  current  ARL 
FedLab  programs  were  awarded  in  January  1996. 
Three  consortia  were  established:  Advanced 
Sensors;  Advanced  Telecommunications  &  Informa¬ 
tion  Distribution;  and  Advanced  &  Interactive  Dis¬ 
plays.  Consortium  membership  ranges  from  6  to  12 
industry  and  university  partners. 

The  Advanced  Sensors  Consortium  (ASC)  conducts 
research  in  support  of  the  next  generation  of  more 
affordable  and  more  capable  Army  sensors  for 
future  combat  systems,  the  individual  soldier,  and 
Army  aviation.  Research  is  grouped  into  the  follow¬ 
ing  major  areas; 

a)  Multi-Domain  Smart  Sensors:  Research  aimed 
at  developing  technology  for  common-aperture, 
multispectral  passive/active  imagers  that  will 
achieve  a  major  performance  improvement 
compared  to  state-of-the-art  infrared  imagery 
systems. 

b)  Muiti-Sensor  Fusion  Automatic  Target  Recogni¬ 
tion:  Research  focused  on  the  combination  of 
information  from  muitiple  sensors  to  provide 
improved  recognition  performance. 

c)  Radar:  Research  to  enable  the  development  of 
compact,  affordable  millimeter-wave  radars  and 
ultra-wideband  radars  for  obscured  target 
detection. 

d)  Signal  Processing:  Research  to  develop  en¬ 
abling  technology  for  both  low-power  individual 


soldier  applications  and  for  compact  high- 
density  tactical  signal  processors. 

e)  Microsensors:  Research  focused  on  advancing 
technology  for  Army  applications  based  on 
micro-electro-mechanical  systems  (MEMS)  and 
other  small,  low-cost,  low-power  sensor  devices. 

In  the  past  four  years,  substantial  progress  has  been 
made  in  many  of  these  areas,  and  technology 
products  are  beginning  to  transition  from  the  Ad¬ 
vanced  Sensors  consortium  to  various  programs 
both  in  and  outside  the  Army.  Examples  include 
multicolor  infrared  cameras  and  networked  micro¬ 
sensors.  High-performance  multicolor  focal  plane 
arrays  have  been  transitioned  to  Army  and  other 
DOD  programs.  Low-power  signai  processing 
techniques  developed  in  FedLab  are  transitioning  to 
unattended  ground  sensor  and  micro-robot  sensor 
programs. 

The  Advanced  Telecommunication  &  Information 
Distribution  Research  Program  (ATIRP)  is  conducting 
research  in  mobiie  wireless  communications  in 
support  of  a  more  deployable,  agile,  and  survivable 
force.  Research  is  grouped  into  the  following  major 
areas; 

a)  Wireless  Battlefield  Digital  Communications: 
Research  focused  on  providing  wireless  digital 
communications  in  an  integrated  architecture  for 
mobiie  warfighters. 

b)  Tactical-Strategic  Interoperability;  Research  to 
provide  seamless  interoperability  and  manage¬ 
ment  of  heterogeneous  networks. 

c)  Information  Distribution:  Research  to  provide 
automated  information  distribution  from  dispar¬ 
ate  sources  on  the  battlefield  with  reliability, 
adaptability,  and  robustness. 

d)  Multimedia  Concepts:  Research  to  provide 
muitimedia  communications  to  enhance  the 
situation  awareness  of  the  warfighter. 

e)  Defensive  Information  Warfare:  Research  to 
provide  survivable  and  secure  tactical  networks 
and  information  warfare  protection  technologies. 

The  ATIRP  consortium  has  successfully  transitioned 
a  number  of  advanced  technologies  to  Army  and 
other  DOD  customers.  Examples  include  a  mobile 
internetworking  protocol  for  the  Army’s  Warfighter 
Information  Network  testbed  and  technologies  to 
enable  reliable,  robust  underwater  acoustic  commu¬ 
nications  for  Navy  submarines. 
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The  Advanced  &  Interactive  Displays  Consortium  is 
conducting  research  in  software,  algorithms,  and 
human  factors  required  for  the  effective  display  and 
presentation  of  information  and  knowledge  to 
warfighters  on  a  broad  variety  of  hardware  platforms. 
Research  is  grouped  into  the  following  major  areas: 

a)  Cognitive  Engineering:  Research  in  database 
management  for  battlefield  visualization  and 
planning,  intelligent  information  processing  for 
visualization,  and  course-of-action  planning. 

b)  Perception-Based  Display  Formats:  Research  in 
improved  multimodal  human-computer  interac¬ 
tion  (including  speech  recognition,  eye-tracking, 
three-dimensional  audio,  and  other  modalities) 
and  augmented  reality  (the  overlay  and  registra¬ 
tion  of  real  and  virtual  worlds). 

c)  Soldier-Centered  Computer  Interface:  Research 
in  the  integration  of  new  human-computer 
interaction  modalities  into  systems  for  human 
factors  evaluation  and  use. 

The  Advanced  &  Interactive  Displays  Consortium  is 
successfully  transitioning  a  number  of  research 
products  to  Army  and  other  DOD  programs.  Several 
innovative  decision  aids  have  been  transmitted  to  the 
Army  Battle  Labs  for  evaluation.  Major  elements  of 
the  multimodal  integrated  display  testbed  have  been 
transferred  to  Army  customers  as  well. 

The  three  FedLab  consortia  have  also  jointly  planned 
and  executed  research  aimed  at  developing  tech¬ 
nologies  for  networks  of  low-cost  unattended  sen¬ 
sors.  Research  supporting  networked  microsensors 
includes  low-power  signal  processing,  sensor  beam¬ 
forming,  point-to-point  communications,  network 
topology,  and  data  and  information  display  tech¬ 
niques.  Research  products  developed  through  this 
collaboration  have  been  evaluated  in  a  series  of  field 
experiments  in  realistic  environments.  Much  of  this 
research  is  transitioning  to  an  Army  applied  research 
program  designed  to  demonstrate  extended  battle- 
space  sensing  for  the  warfighter. 

Future  Plans 

The  current  FedLab  programs  will  conclude  in  mid- 
2001.  Based  on  the  successes  of  these  programs,  it 
has  been  decided  to  initiate  a  new  generation  of 
public-private  partnerships  that  build  on  the  current 
FedLab  programs.  The  new  generation  of  partner¬ 
ships  will  again  focus  on  technologies  where  the 
private  sector  has  a  significant  lead,  and  which  can 


substantially  improve  the  Army’s  ability  to  dominate 
the  full  spectrum  of  future  military  operations. 

The  Chief  of  Staff  of  the  Army  has  outlined  the  new 
Army  Vision  and  his  intention  to  lead  a  transforma¬ 
tion  of  the  Army.  The  elements  of  this  vision  and 
transformation  include  making  the  Army  more 
deployable,  lethal,  and  survivable,  as  well  as  reduc¬ 
ing  its  logistical  footprint.  The  heavy  forces  must 
become  more  deployable  and  the  light  forces  more 
lethal.  It  is  widely  recognized  that  technology  will 
play  a  major  role  in  achieving  this  vision,  both  in  the 
near  and  far  term,  as  the  Army’s  Objective  Force  is 
developed  and  fielded. 

The  next-generation  FedLab  will  address  technolo¬ 
gies  essential  to  fully  realizing  this  vision,  again 
drawing  expertise  from  the  private  sector  where  it 
has  strong  capabilities.  These  technologies  include 
advanced  sensors,  advanced  decision  architectures, 
communications  and  networks,  robotics,  and  power 
and  energy. 

Government  participation  in  the  programs  will  be 
broadened:  other  Army  organizations,  as  well  as 
other  government  agencies,  will  be  able  to  partici¬ 
pate  through  a  Research  Management  Board  (RMB) 
chaired  by  ARL.  Other  government  agency  participa¬ 
tion  on  the  RMB  will  provide  these  agencies  the 
opportunity  to  influence  the  research  programs,  to 
identify  other  applications  for  technologies  devel¬ 
oped  by  the  consortia,  and  (if  appropriate)  to  provide 
funding  to  facilitate  transition  of  those  technologies 
to  customer  programs  and  applications. 

Summary 

The  Army’s  Federated  Laboratory  program  is  a 
successful  example  of  a  true  public-private  sector 
partnership.  Strong  collaborative  relationships 
among  government,  industry,  and  university  re¬ 
searchers  have  been  established  and  are  leading  to 
high-quality  research  and  technology.  The  collabora¬ 
tive  nature  of  these  partnerships  has  greatly  facili¬ 
tated  not  only  the  transfer  of  knowledge  within 
FedLab,  but  also  the  transfer  of  innovative  technolo¬ 
gies  to  Army  and  other  customer  applications.  The 
program  has  been  recognized  by  the  Office  of  the 
Vice  President  with  a  Hammer  Award  as  an  example 
of  reinventing  the  way  the  government  does  busi¬ 
ness.  The  Army  and  ARL  intend  to  build  on  these 
programs  and  initiate  a  new  expanded  FedLab  that 
will  produce  key  technologies  to  enable  the  new 
Army  Vision. 
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Abstract 

The  iterative  process  of  anticipating  the  changing 
environment,  acting  on  likely  courses  of  action,  and 
maintaining  situational  awareness  to  insure 
expectations  remain  aligned  with  reality  lead  an 
organization  to  early  discovery  of  many  of  the  roles 
and  working  practices  required  for  success  in  a 
changing  environment.  This  process  also  leads  an 
organization  into  a  cycle  of  continual  change  and  the 
additional  challenges  that  this  creates. 

Currently,  the  Defense  Community  is  in  a  period  of 
change  known  as  the  Revolution  in  Military  Affairs 
(RMA).  The  RMA  is  the  adoption  of  technology 
advancements  to  change  the  military  operational 
model  to  meet  the  changing  circumstances  of  the 
post-Cold  War  era,  an  era  of  reduced  budgets, 
manpower  reductions  and  increased  commitments, 
in  both  military  operations  and  defense  business  the 
demand  for  change  is  a  constant.  Managing  this 
change  over  time  is  essentially  the  operational  art 
and  campaigning  in  its  warfighting  sense.  One 
school  of  this  period,  the  advocates  of  Network 
Centric  Warfare,  is  studying  the  leading  information 
age  organizations  to  learn  how  these  corporations 
have  adopted  information  age  concepts  and 
technologies  to  leverage  information  and  information 
technology  (IT)  to  create  competitive  advantage. 

In  Net  Ready,  Strategies  for  success  in  the  E- 
conomy,  Amir  Hartman  and  John  Sifonis,  have 
analyzed  the  experience  of  more  than  400 
companies  involved  in  the  Internet  business 
revolution  and  developed  the  concept  of  Net 
Readiness®.  They  have  defined  the  common 
characteristics  found  in  varying  degrees  in  all  these 
corporations.  These  common  characteristics,  the 
pillars  of  Net  Readiness®,  are  leadership, 
governance,  technologies  and  competencies. 

In  the  early  1990’s  Cisco  Systems  was  a  successful 
company  providing  the  hardware  and  software  that 
comprised  80%  of  the  backbone  of  the  Internet. 
However,  there  also  were  numerous  challenges. 

The  organization  had  grown  70%  annually. 

Customer  satisfaction  levels  were  satisfactory,  but 
stagnant,  and  it  was  becoming  difficult  to  find 
enough  skilled  engineers  to  respond  to  customer 
queries  and  to  service  increasingly  complex 
networks.  To  remain  successful  Cisco  needed  to 
find  ways  to  keep  up  with  the  demands  of  the  market 
place  and  the  requirements  of  the  individual 
customer. 


Cisco's  response  to  the  continuing  demand  for 
change  to  meet  these  driving  forces  has  brought  it 
significant  success. 

In  1992  in  response  to  its  rapid  growth  and  the 
opportunity  for  continued  growth  Cisco  started 
development  of  an  Internet  business  model  --  the 
Global  Networked  Business  model.  This  model  uses 
the  network  to  build  interactive,  knowledge-based 
relationships  with  customers  -  e-commerce  and 
customer  service,  with  partners  and  suppliers  - 
supply  chain  management,  and  with  employees  - 
workforce  optimization.  These  solutions  are  Cisco 
Connection  Online  (CCO),  Manufacturing 
Connection  Online  (MCO)  and  Cisco  Employee 
Connection  (CEC).  The  result  of  this  network  centric 
business  model  for  fiscal  year  1999  were  net  sales 
of  $12.2  billion,  $825  million  in  savings  from  Internet 
business  processes,  and  a  per  employee 
productivity  figure  of  $688,000  as  compared  to 
$250,000/employee  in  the  closest  industry  vertical, 
other  Internet  companies.  Additionally,  once  proven 
to  be  a  successful  model,  the  model  itself  became  a 
source  of  growth 

Admittedly  not  one  hundred  percent  transferable, 
Cisco's  success  during  the  1990's  can  provide  an 
experienced-based  insight  for  the  T&E  community 
into  the  process  of  preparing  for  the  "Roles  and 
Working  Practices  in  the  Changing  Environment." 

Introduction  -  The  Budget,  Business  and  Operational 
Challenges 

Since  1989  both  warfighting  and  Operations  Other 
Than  War  have  keep  the  troops  in  the  field, 
consumed  the  service  life  of  equipment  and  depleted 
operational  stocks.  We  have  seen  large-scale 
efforts  such  as  Desert  Shield/Desert  Storm,  Bosnia 
and  Kosovo,  smaller  scale  peacekeeping  missions  in 
East  Timor  and  Somalia,  repeated  deployments  to 
festering  trouble  spots  such  as  Sierra  Leone  and  the 
Congo,  and  numerous  humanitarian  relief  operations 
throughout  the  world. 

Also,  since  1989  the  defense  community  has  been  in 
a  period  of  continuous  change.  The  changes  have 
been  in  all  aspects  of  defense  --  budgets,  manning, 
doctrine,  organization,  and  business  process.  In 
general,  the  trend  lines  for  the  resource  categories 
have  been  down,  while  the  operational  tempo, 
change,  and  consumption  trend  lines  have  been  up. 
This  has  created  a  mismatch  of  commitment  to 
resources. 
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In  addressing  the  Australian  problem,  Dr.  Allen 
Hawke,  Secretary  of  the  Australian  Department  of 
Defense  succinctly  stated  the  problem  faced  by 
many  nations. 

"The  irony  of  our  professional  military 
performance  in  East  Timor  is  that  it  masks 
the  reality  we  face.  Australia's  national 
security  is  challenged  by  a  convergence  of 
financial,  management,  planning  and 
strategic  pressures.  The  Australian  Defence 
Organisation’s  ability  to  present  a  range  of 
capability  and  military  response  options  to 
government  will  be  severely  constrained  if 
these  combined  pressures  are  left 
unchecked.  This  crisis,  which  has  been 
building  over  the  last  decade,  has  now  come 
to  a  head  due  to  increased  personnel  costs 
and  the  costs  of  expanding  and  re-equipping 
the  capabilities  of  the  ADF  [i] 

Less  the  issues  of  funding,  the  impact  the  Internet 
economy  on  business  is  not  dissimilar  from  the 
circumstances  in  defense.  Businesses  have  had  to 
cope  with  change  or  see  their  future  jeopardized. 

The  requirements  for  and  the  tempo  of  change  in  are 
intense. 

In  their  study  of  Internet  business.  Net  Ready, 
Strategies  for  Success  in  the  E-conomy,  Hartman 
and  Sifonis  map  the  change  from  the  Traditional 
economy  to  the  E-conomy,  (Figure  1).  The  E- 
conomy  being  "the  virtual  arena  in  which  business 
actually  is  conducted,  value  created  and  exchanged, 
transactions  occur,  and  one-to-one  relationships 
mature."  [ii] 

[i]  Dr.  Allen  Hawke,  Secretary  of  the  Department  of 
Defence,  Australia  from  a  paper  based  on  an 
Address  to  the  Royal  United  Services  Institute  for 
Defence  Studies,  Victoria,  Australia,  27  April  2000. 

[ii]  Amir  Hartman  and  John  Sifonis,  Net  Ready, 
Strategies  for  success  in  the  E-conomy  (New  York: 
McGraw-Hill,  1999)xviii. 
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Figure  1 .  The  Change  from  the  Traditional 
economy  to  the  E-conomy  [iii] 

The  current  leading  corporations  of  the  E-conomy 
have  not  just  changed  but  have  done  so  in  Internet 
time.  (Operational  cycles  measured  in  Internet  years 
where  three  months  equates  to  a  normal  business 
year.)  Also,  they  have  established  change  as  the 
organizational  norm. 

These  changes  in  business  and  their  impacts  are 
relevant  to  the  defense  community. 

A  projection  by  the  Giga  Information  Group  states, 

"In  2002,  the  almost  $1  trillion  in  predicted  global 
Internet  commerce  revenues,  will  also  generate  cost 
savings  of  nearly  $1 .25  trillion,  producing  even 
greater  profit  margins  than  revenue  growth,  dollar  for 
dollar,  [iv]  The  take  away  point  is  that  using  Internet 
technologies  will  reduce  or  eliminate  many  costs  by 
improving  core  business  processes.  This  means  the 
Internet  is  more  than  just  a  new  way  to  sell  products 
and  services,  it’s  a  way  to  efficiently  run  a  business, 
resulting  in  significant  cost  savings  that  add  to  an 
organization's  bottom  line. 

[iii]  Ibid.  6 

[iv]  Extract  of  Cisco  Systems,  Inc.,  Corporate 
Overview  quoting  from  the  Giga  Information  Group, 
Cambridge  MA,  1999. 
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If  only  for  the  bottom  line  benefits  of  networked 
business  process,  Defense  should  leverage  the 
business  process  changes  of  Internet  commerce. 
Given  the  funding  shortfalls  in  defense,  it  is  hard  to 
argue  against  implementing  changes  that  are 
estimated  to  generate  $1 .25  trillion  in  savings.  Put  in 
historical  terms,  Cisco’s  Global  Network  Business 
Model  has  produced  measurable  savings  of  $325 
million  in  1997,  $525  miliion  in  1998  and  $825  million 
in  1999. 

Defense's  businesses  also  have  the  challenge  of 
adopting  practices  that  are  compatible  with  their 
major  business  partners.  Besides  the  traditional 
defense  contractors  that  are  increasingly  vested  In 
commercial  business  as  a  source  of  growth,  the 
deployment  of  commercial-off-the-shelf  technology 
places  defense  squarely  in  the  larger  market  place  of 
business-to-business  commerce.  In  this  market  the 
influence  of  defense's  "power  of  the  purse"  is  greatly 
reduced.  It  is  an  arguable  point  that  the  failure  of 
defense  to  adopt  commercial  business  practices 
may  cause  businesses  to  ask  the  question,  "Is  it 
better  to  forgo  defense  business  than  to  deal  with 
defense  on  a  by-exception  basis  from  the  rest  of  my 
business  partners?" 

Even  in  major  weapons  system  acquisitions  the 
business-defense  relationship  has  moved  from 
favored  customer  towards  that  of  consumer.  The  US 
Navy's  acquisition  of  DD  21 ,  the  next  generation 
surface  combatant,  is  a  competition  between  two 
consortia  that  have  been  challenged  to  design,  build, 
and  maintain  the  ship  for  its  service  life  while  making 
a  profit.  The  Navy  should  expect,  better  yet 
demand,  that  the  business  system  built  around  this 
ship  is  also  state-of-the-art  in  business  process.  In 
selecting  the  winner  the  US  Navy  must  fili  the  role  of 
well-informed  consumer.  The  Acquisition  Warrior 
war  games  for  DD  21  indicate  the  Navy  has 
recognized  the  need  for  change  in  acquisition. 

As  the  defense  business  broadens  to  commercial 
corporations  and  as  the  defense  industrial  base 
becomes  increasingly  a  community  of  commercial- 
defense  hybrids,  then  defense  business  process 
must  move  to  the  internet-based  business-to- 
business  model.  The  failure  of  the  defense  to  adopt 
these  practices  will  result  in  defense's 
marginalization  from  the  mainstream  business 
community. 

The  operational  challenges  are  similar.  Military 
forces  must  execute  with  a  speed  of  command  and 
technology  level  "better"  than  their  adversary  or 
accept  the  risk  of  "defeat"  by  an  adversary  who 
seizes  the  initiative  by  creating  an  asymmetric 
advantage  through  network  centric  operations  or  a 
leap-ahead  technology. 


The  requirement  for  operational  and  technological 
change  is  collectively  known  as  the  Revolution  in 
Military  Affairs  (RMA)  —  the  drive  to  harness 
technology  to  ultimately  bring  about  fundamental 
conceptual  and  organizational  change  In  defense. 

The  RMA  is  characterized  by  leap-ahead  advances 
in  information  technologies:  by  having  no  definitive 
linear  process;  and  by  its  requirement  for 
experimentation  both  to  understand  the  potential 
contributions  of  emerging  technologies  and  to 
develop  innovative  operational  concepts  to  harness 
these  new  technologies,  [v] 

In  the  United  States'  Joint  Vision  2010  (JV2010) 
information  superiority  is  the  capstone  requirement 
for  fueling  achievement  of  the  operation  concepts  of 
Dominant  Maneuver,  Precision  Engagement, 

Focused  Logistics  and  Force  Protection,  [vi] 

Another  term  frequently  encountered  in  this  lexicon 
of  operational  change  is  Network  Centric  Warfare 
(NCW).  Network  Centric  Warfare  is  defined  as;] 

An  information  superiority-enabled  concept 
of  operations  that  generates  increased 
combat  power  by  networking  sensors, 
decision  makers,  and  shooters  to  achieve 
shared  awareness,  increased  speed  of 
command,  higher  tempo  of  operations, 
greater  lethality,  increased  survivability,  and 
a  degree  of  self-synchronization.  In 
essence,  NCW  translates  information 
superiority  into  combat  power  by  effectively 
linking  knowledgeable  entities  in  the 
battlespace."  [vii] 

Vice  Admiral  Cebrowski,  President,  US  Naval  War 
College  comments, 

"Network  Centric  Warfare  is  increasingly 
seen  as  the  military  expression  of  the 
information  age.  More  specifically.  Network 
Centric  Warfare  enables  promising 
alternative  concepts  of  Command, 

Command  relations,  and  Command  and 
Control  processes,  [viii] 

[v]  William  S  Cohen,  U.  S.  Secretary  of  Defense,  Annual 
Report  to  the  President  and  the  Congress  1998, 
Washington,  DC,  117. 

[vi]  Chairman  of  the  U.  S.  Joint  Chiefs  of  Staff,  Joint  Vision 
2010,  Washington,  DC,  1. 

[vii]  David  S.  Alberts,  et  al,  Network  Centric  Warfare, 
Developing  and  leveraging  Information  Superiority,  C4ISR 
Coop.  Research  Program,  Wash.,  DC,  2. 

[viii]  VAdm.  Arthur  K  Cebrowski,  USN,  President,  US 
Naval  War  College  in  his  Convocation  Address  for 
Academic  Year  1998,  Newport  Rl,  Aug  1998. 
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In  the  chapter  "Information  Age  Organizations"  in 
Network  Centric  Warfare,  Developing  and 
Leveraging  Information  Superiority,  the  authors 
note  that  commercial  "organizations  that  have  been 
able  to  fully  leverage  the  power  of  information  and 
information  technology  (IT)  to  develop  competitive 
advantage  have  dominated  their  competitive 
domains."  They  also  make  note  "that  business  is  not 
warfare  ...  But  to  dismiss  a  potentially  rich  source  of 
hypotheses  ...  is  as  foolish  as  it  is  unnecessary." 

In  this  recognition  of  the  changes  in  the  networking 
industry,  they  make  the  point  that  defense  is 
between  1-  and  10-percent  plus  of  the  $600  billion- 
plus  Information  Technology  market.  Also,  it  is  this 
sector  where  the  case  studies  and  insights  are 
available  to  help  defense  become  an  information  age 
organization,  [ix] 

The  Operational  Art  -  Change  requires  leadership 
and  vision.  It  also  happens  over  time. 

The  operational  level  is  the  link  between  tactical 
success  in  combat  and  what  ultimately  in  war  is 
success  “  attaining  the  aims  of  policy.  The 
operational  art  is  the  ability  to  envision  the  actions 
required  to  achieve  the  strategic  end  state  by  using 
the  available  forces,  time,  space  and  resources  to 
create  advantageous  position  from  which  to  enter  a 
series  of  meaningful  battles.  This  series  is  a 
campaign  and  it  is  the  primary  tool  of  operational 
warfare,  [x] 

In  his  discussions  of  Network  Centric  Warfare  VAdm. 
Cebrowski  states  "  Furthermore,  there  is  the 
fundamental  question  of  what  the  commander 
commands  -  forces,  information  services,  or  key 
processes?"  [xi] 

Likewise  in  business,  the  question  of  what  a 
"commander  commands"  is  critical  to  the  changing 
nature  of  business  and  the  business  leader's 
success  in  orchestrating  change.  It  is  a  question 
answered  by  Cisco  in  developing  its  Global 
Networked  Business  Model  and  subsequently  more 
formally  in  Net  Ready  -  Strategies  for  Success  in 
the  E-conomy.  As  defined  in  Net  Ready,  Net 


[ix}  David  S.  Alberts,  et  al.  Network  Centric  Warfare, 
Developing  and  leveraging  Information  Superiority, 
C4ISR  Coop.  Research  Program,  Wash.,  DC,  23  - 
27. 

[x]  U.S.  Marine  Corps,  Fleet  Marine  Force  Manual  1- 
1 ,  Quantico,  VA. 

[xi]  VAdm.  Arthur  K  Cebrowski,  USN,  President,  US 
Naval  War  College  in  his  Convocation  Address  for 
Academic  Year  1998,  Newport  Rl,  Aug  1998 


Readiness®  is  "a  combination  -  unique  to  each 
Company  —  of  four  drivers  that  enables  enterprises 
to  deploy  high-impact  Web-enabled  business 
processes  that  are  focused,  accountable  and 
measurable."  The  drivers  are  leadership, 
governance,  competencies  and  technologies. 

In  the  Net  Ready  businesses,  the  question,  'Who 
commands  what?"  is  addressed  by  the  leadership 
and  governance  pillars.  While  "available  forces, 
time,  space  and  resources"  may  be  equated  to  the 
competencies  and  technology  pillars. 

The  fact  that  the  end  state  of  Net  Readiness  is  to 
develop  processes  that  are  focused,  accountable 
and  measured  again  make  studying  Cisco's  change 
from  a  traditional  manufacturing  company  to  a 
network  centric  corporation  a  case  study  of  interest 
T&E  community.  I've  mentioned  the  savings 
realized  from  1997  to  1999  by  Cisco's  network 
centric  business  model.  Figure  2  depicts  Cisco's 
Breakaway  performance  against  its  Traditional 
Networking  Industry  Competition  since  commencing 
its  campaign  of  Internet  business.  It's  a  testimony  of 
success  through  the  adoption  of  a  network  centric 
business  model  where  information  availability 
creates  value  add. 

($B) 


Source:  Bloomberg 

Notes:  Dates  are  calendar  years,  Numbers  are 
annualized  from  most  recent  reports 

Figure  2:  Cisco's  Breakaway  performance  against  its 
Traditional  Networking  Industry  Competition  [xiii] 

[xii]  Amir  Hartman  and  John  Sifonis,  Net  Ready, 
Strategies  for  success  in  the  E-conomy  (New  York: 
McGraw-Hill,  1999)  3. 

[xiii]  Extract  of  Cisco  Systems,  Inc.,  Corporate 
Overview  quoting  Bloomberg,  Mar  00  version. 
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The  Four  Pillars  of  Netreadiness®  -  Leadership, 
Governance.  Competencies  and  Technologies. 

In  their  study  of  E-conomy  successes,  Hartman  and 
Sifonis  studied  400  companies  and  culled  the 
success  traits  in  companies  as  Cisco  Systems, 
Microsoft,  Charles  Schwab,  Dell  Computers,  eBay, 
and  Amazon.com.  They  found  that  there  are  in  an 
infinite  variety  of  permutations  of  leadership, 
governance,  competencies  and  technologies  that  are 
consistently  displayed  by  the  most  successful 
companies,  [xiv] 

Originally  examined  with  a  singular  vision  in  a  Cisco 
White  Paper  "Global  Networked  Business  Approach 
and  Architecture  [xv],"  the  in-depth  analysis  of  Net 
Readiness  broadens  the  horizon  of  source  data  and 
gives  greater  academic  credence  to  defining  the 
requirements  for  success  in  the  E-conomy.  In  this 
paper  I  can  only  give  a  brief  description  of  the  pillars. 

I  hope  however  it  encourages  further  study  of  the 
drivers  for  implementing  network  centric  change. 

Leadership.  Most  of  the  success  stories  in  the  E- 
conomy  have  a  recognized  leader  at  the  helm  but 
the  leadership  pillar  is  more  than  the  persona  of  a 
key  individual.  Leadership  is  a  pervasive  quality  of 
successful  network-centric  businesses.  Every 
member  of  the  organization  must  be  a  leader.  From 
the  CEO  on  down,  each  team  member  must  think 
and  act  in  E-conomy  terms,  use  E-business  tools, 
and  be  accountable  in  clearly  measurable  ways  for 
pursuing  the  network  centric  end-state.  The 
overarching  leadership  message  is  that  network 
centric  culture  runs  from  top  to  bottom,  and  everyone 
is  empowered  to  support  and  propagate  business 
focused  on  that  culture.  "The  leader’s  job  is  to 
promote  this  vision  by  personal  example."  [xvi]  As 
they  did  at  the  macro-level  of  the  old  and  new 
economies,  Hartman  and  Sifonis  have  mapped  the 
Qualities  of  Net  Ready  leadership  of  the  Traditional 
CEO  and  the  Net  Ready  CEO,  Figure  3. 

[xiv]  Amir  Hartman  and  John  Sifonis,  Net  Ready, 
Strategies  for  success  in  the  E-conomy  (New  York: 
McGraw-Hill,  1999)3-4. 

[xv]  Cisco  Systems,  Inc.,  Building  a  Global 
Networked  Business,  San  Jose  CA,  1998. 

[xvi]  Amir  Hartman  and  John  Sifonis,  Net  Ready, 
Strategies  for  success  in  the  E-conomy  (New  York: 
McGraw-Hill,  1999)5. 
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Figure  3:  Qualities  of  Net  Ready  leadership  of  the 
Traditional  CEO  and  the  Net  Ready  CEO  [xvii] 

As  you  read  this  list  I've  little  doubt  you  found 
yourself  questioning  the  applicability  of  some  of  traits 
to  defense  executives.  This  is  yet  another  challenge 
for  the  defense  community  in  this  time  of  change.  Its 
leaders  and  managers  must  have  the  intellectual 
flexibility  to  cull  from  the  wide  variety  of  findings 
about  business  process  change  the  points  that  are 
useful  even  when  the  explanations  and  examples 
have  nothing  to  do  with  defense.  Defense  leaders 
must  be  able  to  envision  the  actions  required  to 
achieve  the  strategic  end-state.  They  must  practice 
the  operational  art.  Defense  leaders  know  "that 
business  is  not  warfare"  as  well  as  the  fact  that 
defense  business  is  not  commercial  business.  The 
point  made  about  Network  Centric  warfare  earlier 
remains,  not "...  to  dismiss  a  potentially  rich  source 
of  hypotheses,"  because  it  is  not  a  one-to-one 
mapping. 

Governance.  This  determines  the  nature  of  the 
relationships  within  and  outside  the  formal 
organization.  Think  of  the  US  Department  of 
Defense  as  you  read  the  following. 

"A  modern  enterprise  consists  of  multiple 
business  units  and,  in  many  instances, 
overlapping  business  functions, 
administrative,  manufacturing,  marketing, 
support  and  so  on.  To  successfully  drive  E- 


[xvii]  Ibid.  6. 

[xviii]  David  S.  Alberts,  et  al.  Network  Centric 
Warfare,  Developing  and  leveraging  Information 
Superiority,  C4ISR  Coop.  Research  Program, 
Wash.,  DC,  24. 
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business,  an  organization  must  have  a 
mechanism  that  enables  it  to  cross  all  these 
functions,  taking  responsibility  for  business 
initiatives  spanning  different  operating  units." 

[xix] 

Successful  network  centric  corporations  have 
defined  their  governance  and  operational 
frameworks  to  handle  continuous  and  rapid  change 
by  focusing  on  four  core  disciplines:  the  governance 
model,  the  decision  process,  the  policies  and 
standards  for  implementing  change,  and  the  goals 
and  metrics  defining  performance  objectives  and 
their  measurement. 

Governance  lessons  that  standout  from  the  studies 
of  Hartman  and  Sifonis  include: 

■  Establish  cross-functional  teams. 

■  Demand  near-term  E-business  results. 

■  Actively  promote  the  use  of  E-business 
applications. 

■  Make  E-business  a  business-driven  activity,  not 
a  technology-driven  staff  function. 

•  Make  E-business  funding  decisions  resemble  all 
business  funding  decisions. 

•  Establish  a  cross-functional  governance  council 
with-business,  technology  and  evangelical 
components 

■  Make  IT  take  on  a  free  market  fulfillment  role. 

[XX] 

Technology.  This  is  not  technology  in  the  narrow 
sense  of  protocols  and  individual  standard  but 
technology  from  an  operational  perspective.  The 
successful  Net  Ready  companies  have  developed 
architectures  that  are  robust  and  comprehensive, 
that  enable  them  to  develop  and  deploy  new 
applications  without  the  having  to  rework  the 
infrastructure  -  the  network,  data  security  and 
databases,  with  each  change  in  the  way  they  do 
business  in  a  segment  of  their  business. 

[xix]  Amir  Hartman  and  John  Sifonis,  Net  Ready, 
Strategies  for  success  in  the  E-conomy  (New  York: 
McGraw-Hill,  1999)  13. 

[xx]  Ibid.  18. 
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Figure  4:  Technical  Architecture 

Competencies.  Just  as  information  superiority  is  an 
enabler  for  the  operational  concepts  of  Joint  Vision 
2010,  competencies  are  the  cement  of  governance, 
leadership  and  technologies  for  Net  Readiness®. 
Competencies  are  defined  as  "the  ability  to  navigate 
among  the  these  values....  Competencies  determine 
the  way  Net  Ready  organizations  respond  to 
changes  in  the  world,  exploit  available  resources  and 
opportunities  and  accommodate  themselves  to 
emerging  realties."  [xxi] 

Now,  with  this  framework  let’s  briefly  examine  Cisco 
Systems. 

Cisco's  Network  Centric  Campaign 

Since  1992  Cisco  has  been  a  company  in  change. 
These  changes  were  business  critical  changes  to 
meet  the  challenges  of  scale,  the  speed  of  technical 
innovation  of  the  networking  industry,  the  need  to 
stream  line  business  process,  and  the  need  for 
communications  with  customers,  supplier,  partners 
and  employees. 

Corporate  business  discussions  use  the  same  terms 
as  the  military  -  "vision",  "strategy",  "tactics",  and 
"risk  management." 

[xxi]  Ibid.  19 
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At  Cisco  the  strategic  end-state  is  hardly  a  modest 
one.  The  corporate  vision  is  "Changing  the  way  we 
work,  live,  play  and  learn."  The  mission  derived  from 
this  is  "Shape  the  future  of  the  Internet  by  creating 
unprecedented  value  and  opportunity  for  our 
customers,  employees,  investors  and  ecosystem 
partners."  The  Cisco  Global  Networked  Business 
Model  highlights  Cisco's  ability  to  connect  strategy  to 
tactics  -  the  operational  level.  As  for  the  art  of 
campaign,  Cisco's  ability  to  sustain  a  continuum  of 
change  to  create  business  advantage  is  best 
demonstrated  by  both  the  savings  and  breakaway 
performance  mentioned  above. 

The  discussions  in  Network  Centric  Warfare 
highlight  the  requirement  for  the  co-evolution  of 
organizations  and  process  while  exploiting  new 
technology.  More  importantly  it  notes  that 
organizations  have  different  time  constraints  with 
respect  to  change,  [xxii] 

In  "Net  Readiness  at  Cisco  Systems"  Cisco's  co¬ 
evolution  has  been  characterized  as  follows: 

"We  believe  the  Cisco  story  reveals  the 
impact  that  a  sustained  dedication  to  the 
principles  of  Net  Readiness®  can  bring  to  an 
organization  willing  to  accept  its  disciplines. 
But  Cisco's  success  was  not  inevitable.  It 
would  be  a  mistake  to  assume  that  the 
seamless  electronic  processes  that  enable 
Cisco  emerged  out  of  a  well-ordered, 
analytical  process.  What  really  happened  is 
that  a  lot  of  dedicated  Cisco  people 
responded  to  a  clarity  of  vision  with  a  large 
number  of  initiatives,  many  of  which  proved 
to  be  quite  effective.  More  important, 

Cisco's  evolvement  didn't  happen  overnight. 
In  fact,  the  Cisco  story  is  the  product  of  five 
years  of  effort,  millions  of  dollars  in 
investment,  and  hundreds  of  projects,  not  all 
of  which  proved  sustainable."  [xxiii] 

The  methodology  for  Cisco  has  been  a  series  of 
tactical  changes  and  experiments  within  a  visionary 
framework  centered  on  customer  service  and 
creating  value-add  for  all  Cisco's  partners. 


Of  particular  note  at  Cisco  is  the  demand  by  and 
tolerance  of  Cisco's  leadership  that  risk  taking  is  the 
expected  course  of  action.  Cisco  is  not  a  not-born- 
on-the-web  company  but  it  is  a  company  that 
recognized  very  early  the  strategic,  operational  and 
tactical  implications  of  its  market  and  business 
environment.  Change  at  Cisco  isn't  a  function  of  a 
"big  bang"  that  attempted  to  resolve  all  the  issues.  It 
is  a  series  of  steps  that  encourages  risk,  rewards 
success,  and  creates  a  community  that  accepts  that 
not  all  projects  are  successful. 

As  mentioned  previously,  Cisco's  challenges  in  the 
early  90s  were  the  drivers  for  its  network  centric 
business  model.  At  that  point  Cisco  needed  to  find  a 
way  to  scale  its  operations  in  the  face  of  explosive 
growth  in  the  networking  industry.  It  could  not  rely 
on  finding  enough  skilled  engineers  to  build  and 
service  its  products  and  its  customer's  questions. 
Customer  satisfaction  levels,  though  respectable, 
were  stagnant.  As  customer  satisfaction  is  its  major 
metric  of  success,  the  company  had  to  find  a  way  to 
raise  the  satisfaction  levels  of  its  expanding 
customer  base.  Additionally,  for  continued  growth  it 
needed  to  focus  on  it's  core  competence  -  product 
design  and  innovation,  not  on  fielding  a  team  to 
answer  questions  and  provide  “build  to  order" 
products  for  customers.  Consequently  Cisco  also 
decided  to  outsource  manufacturing  to  a  select 
group  of  partners.  The  challenge  here  was  ensure 
consistently  high  product  quality  across  its  supplier 
base. 

The  chosen  course  of  action  was  to  use  “Internet 
Business  Solutions”  -  leveraging  the  Internet  and 
networked  applications  to  allow  customers  and 
employees  access  to  self-service  tools,  and  to 
automate  repetitive  manual  activities.  Cisco's  first 
step  was  an  electronic  bulletin  board  for  customers. 
It  was  introduced  in  1992  to  post  tech  notes,  provide 
software  upgrades  and  reveal  information  on  its 
software  bugs.  Shortly  after  Mosaic  was  introduced, 
Cisco  allowed  registered  users  with  a  support 
contract  access  to  limited  online  support  features. 
You've  seen  the  overall  Breakaway  performance 
chart  (Figure  2).  Figure  5  shows  Customer 
Satisfaction  on  a  5.0  scale  over  the  same  period. 


[xxii]  David  S.  Alberts,  et  al.  Network  Centric 
Warfare,  Developing  and  leveraging  Information 
Superiority,  C4ISR  Coop.  Research  Program, 
Wash.,  DC,  2  &  4. 

[xxiii}  Amir  Hartman  and  John  Sifonis,  Net  Ready, 
Strategies  for  success  in  the  E-conomy  (New  York: 
McGraw-Hill,  1999)  239. 
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Figure  5  Customer  Satisfaction  [xxiv] 


As  seen  in  Figure  4,  Technical  Architecture,  the 
technologies  pillar  allows  for  deployment  of 
prerequisite  systems,  enabling  technologies  and 
networked  applications.  Figure  6  is  a  table  of  the 
Capabilities  Architecture  for  Cisco’s  Technical 
Information  Center,  and  the  series  of  figures  at  the 
end  of  the  paper  show  how  Cisco  built  the  capability 
into  its  Internet-based  solutions  over  time.  In  this 
case  the  Cisco  Connection  Online  (CCO),  our  link  to 
our  customers. 
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Beyond  the  revenue  and  customer  satisfaction 
measurements,  the  "box  score"  for  Cisco's  online 
customer  care  solutions  is 

•  70%  of  support  calls  are  resolved  by  a  visit  to 
our  website. 

•  $86M  saved  with  electronic  downloads, 

•  $107M  saved  with  online  configuration  and 
documentation, 

•  $40M  in  products  sold  online  each  day,  and 

•  87%  of  orders  placed  online. 

Similar  results  can  be  found  in  the  Cisco 
Manufacturing  Online  solution  for  working  with  our 
partners  in  the  supply  chain,  logistics  and  contract 
manufacturing  area,  and  our  employees  through  the 
Cisco  Employee  Connection.  The  Manufacturing 
box  score  is 

•  70%  cut  in  order  cycle  time  (6-8  weeks  to  1-3 
weeks) 

•  $175M  savings  in  annual  operating  costs 

•  45%  inventory  reduction  for  Cisco  and  partners 

•  Time-to-volume  cut  by  25%;  $270M  annual 
revenue  contribution 

While  the  Cisco  Employee  Connection  is  as 
impressive  with 

•  Self-service  administration  and  information 
resources  for  20,000  employees, 

•  Two  auditors  and  2-day  reimbursement  for 
16,000  U.S.  employees, 

•  Purchasing,  recruiting,  stock  options  and  nearly 
all  employee  services  handled  via  the  intranet, 

•  40-60%  cost  savings  vs.  instructor-led  training, 

•  80%  of  sales  and  technical  training  on  line, 

•  1 00%  of  Cisco  sales  staff  to  use  e-learning  for 
training  in  FY  2000,  and 

•  E-iearning  available  anytime,  with  anywhere 
access  and  full  training  accountability. 

As  stated  previously  the  impact  of  Cisco's  Internet 
solutions  for  1999  were  $825,000,000.  Figure  7 
shows  the  bottom  line  results  of  the  changes  -  The 
Financial  Benefit  by  Internet  Business  Solution. 


Figure  6:  The  Capabilities  Architecture  for  Cisco's 
Technical  Information  Center 

[xxiv]  Extract  of  Cisco  Systems,  Inc.,  Corporate 
Overview  quoting  Bloomberg,  Mar  00  version. 
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Financial  Benefit 

Customer  Care 

$269,000,000 

Internet  Commerce 

$57,000,000 

Supply  Chain  Management 

-  Cost  benefits 

-  Profit  contribution 

$444,000,000 

$175,000,000 

$269,000,000 

Workforce  Optimization 

$55,000,000 

Total  Financial  Impact 

$825,000,000 

Figure  7.  Financial  Impact 


Of  the  many  points  to  be  drawn  from  Cisco's 
success  in  the  E-conomy,  Cisco's  governance  model 
must  be  considered  as  one  of  the  significant 
contributors. 

Information  Technology  (IT)  funding  is  based  on  a 
free-market  fulfillment  model  called  the  Client 
Funded  Project  model.  The  model  is  focused  on  a 


teaming  approach  between  the  business  units  and 
the  IT  department  Figure  8. 

Figure  8.  The  Business  -  IT  team  [xxv] 

The  management  philosophy  is  built  around  four 
major  ideas; 

■  Cisco's  business  units  drive  the  applications  to 
be  deployed. 

■  Business  units  within  Cisco  consider  the  cost  of 
deploying  applications  as  a  cost  of  doing 
business.  The  costs  are  assigned  to  the 
business  unit  that  gains  the  advantage  and  only 

[xxv]  Global  Network  Business  Approach  and 
Architecture  White  Paper,  (San  Jose,  CA:  Cisco 
Systems,  1998).  8.v 


when  benefit  cannot  be  directly  attributed  is  the 
cost  treated  as  corporate  general  and 
administrative  expense. 

■  Business  and  IT  teams  implement  applications 
together  and  their  performance  is  based  on  a 
team  evaluation. 

■  The  Chief  Information  Officer  strictly  enforces 
standards.  This  avoids  the  cost  of  duplicate  or 
unnecessary  spending  created  if  a  business  unit 
were  to  deploy  their  own  unique  technology  or  a 
standard  that  is  incompatible  with  the  rest  of 
Cisco,  [xxvi] 

Incompatibility  is  unacceptable.  An  imperative  of  the 
networked  business  is  that  no  one  owns  information, 
they  have  the  responsibility  to  maintain  its  currency 
so  the  best  and  latest  information  is  available  to 
whoever  needs  the  information  to  conduct  Cisco's 
business. 

Summary.  In  the  discussions  of  "Shaping  T&E  to 
Support  the  Evolution  of  Future  Defence:  Changing 
military,  government  and  industry  roles  to  meet  new 
challenges"  Cisco's  story  offers  many  lessons.  They 
run  the  gamut  from  the  strategy,  operations  and 
tactics  for  implementing  change  over  time,  through 
the  dynamics  of  the  change  process,  to  providing 
examples  of  what  are  the  business  processes  of 
network  centric  operations. 

Although  not  one  hundred  percent  applicable, 

Cisco's  successful  series  of  changes  during  the 
1990's  is  an  experienced-based  insight  into 
changing.  To  again  quote  the  authors  of  Network 
Centric  Warfare "  But  to  dismiss  a  potentially  rich 
source  of  hypotheses  ...  is  as  foolish  as  it  is 
unnecessary"  [xxvii] 

The  frustration  is  that  there  are  too  few  pages  to  give 
more  than  a  sampling  of  the  material  and  the 
selection  of  the  samples,  even  for  a  community  of 
interest  such  as  the  Test  and  Evaluation  Community, 
is  at  best  a  process  of  trial  and  error. 

One  of  the  first  things  a  new  Cisco  employee  hears 
from  his  counterparts  as  he  or  she  starts  to  explore 
the  company's  products,  set-up  their  benefits,  or 
arrange  for  a  corporate  credit  card  is  "its  on  the 
web!"  I'll  end  with  that  idea,  much  of  what's  in  this 
paper  about  Cisco  is  on  the  web  at  www.cisco.com. 

[xxvi}  Amir  Hartman  and  John  Sifonis,  Net  Ready, 
Strategies  for  success  in  the  E-conomy  (New  York: 
McGraw-Hill,  1999)239. 

[xxvii]  David  S.  Alberts,  et  al.  Network  Centric 
Warfare,  Developing  and  leveraging  Information 
Superiority,  C4ISR  Coop.  Research  Program, 

Wash.,  DC,  2&4. 
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Cisco's  Technical  information  Center  Phase  II,  April  1996 
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JAGUAR  NEW  MODEL  LAUNCH  PROCESSES 


Ashley  Emerson 
Jaguar  Cars  Ltd 
Coventry  England 


Abstract 

The  introduction  of  the  XJ  40  series  of  Cars  in  the  mid-eighties  saw  a  low  point  in  Jaguars 
product  launch  performance,  with  the  wrong  product,  poorly  defined,  late  and  with  disastrous 
Quality  and  Reliability.  There  followed  several  years  of  updates,  which  attempted  to  bring  the  car 
up  to  the  level  of  the  competition.  This  action  may  have  been  successful  in  different  economic 
circumstances,  but  in  a  recession  it  led  to  massive  losses  and  the  eventual  take  over  by  Ford. 

After  the  take-over  major  improvements  were  made  to  product  quality,  working  practices  and 
manufacturing  capability.  The  production  sites  at  Browns  Lane  and  Castle  Bromwich  gained  the 
Ford  Q1  Quality  award  and  investments  were  made  in  new  production  facilities.  The  next  step  was 
to  update  the  product;  based  on  past  performance  this  would  be  a  major  risk. 

The  product  was  to  be  a  major  facelift  to  the  saloon  range;  a  successful  launch  was  internally 
regarded  as  a  survival  issue  as  failure  would  surly  mean  that  Ford  would  withdraw  its  financial 
support.  It  was  with  the  scenario  that  failure  would  lead  to  Jaguar  ceasing  to  exist  as  a  pseudo 
independent  entity  the  launch  was  planned. 

The  best  way  forward  was  deemed  to  be  to  adopt  a  whole  new  launch  philosophy,  which 
encompassed  the  best  practices  from  within  Ford.  This  was  done  while  maintaining  the  leverage 
gained  from  a  relatively  small  company  which  can  retain  a  nimbleness  of  process  and  drive  out 
the  functional  chimneys  that  often  dominate  larger  organisations. 

This  Launch  Process  was  successfully  deployed  on  the  95  Model  Year  XJ  series,  codenamed  X300, 
which  proved  to  be  the  most  successful  Jaguar  Launch  to  that  date.  Subsequent  refinements  have 
been  made  and  it  is  the  process,  which  is  a  result  of  those  refinements,  which  is  in  use  today  and 
will  be  the  main  topic  of  the  discussions  that  follow. 
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Jaguar  New  Model  Launch  Processes 


Introduction 

With  the  successful  introduction  of  the  Jaguar  S-Type  and  the  planned  launch  of  the  X400,  the 
company  finds  itself  in  a  much-changed  environment  from  that  which  greeted  the  X300  launch  in 
1994. 

Product  quality  is  no  longer  a  source  of  competitive  advantage.  It  is  a  prerequisite  for  sustaining  a 
position  in  the  market.  The  introduction  of  advanced  quality  tools  and  techniques  were  therefore 
critical  to  the  success,  of  the  programme. 

In  1994,  we  were  seen  as  a  quaint  British  Marque  producing  low  volume  interesting  and  stylish 
cars  that  deserved  praise  for  its  efforts  but  was  not  really  a  serious  competitor  to  either  BMW  or 
Mercedes  Benz.  This  gave  us  the  enviable  position  of  having  a  very  forgiving  World  Press  and 
competitors  who  were  too  preoccupied  with  each  other  to  be  concerned  with  Jaguar.  With  annual 
volumes  planned  to  rise  above  200,000  units,  we  are  now  seen  as  a  serious  car  company  and 
treated  as  such  with  much  more  critical  appraisals  and  competitor  activity. 

The  next  new  Jaguar  Car  must  demonstrate  the  highest  standards  of  excellence  in  British 
Engineering  and  represent  a  further  significant  advance  for  Jaguar,  building  on  the  foundation 
established  for  the  XJ6  saloon  car  project  launched  in  the  mid-nineties  and  re-enforced  in 
subsequent  Product  Launches. 

Jaguar’s  engineering  expertise,  working  in  close  collaboration  with  its  suppliers,  has  enabled 
world  class-leading  products  to  be  launched,  which  demonstrate  all  the  attributes  of 
“Manufacturing  Excellence’. 

The  Company’s  competitive  performance  in  world-wide  markets  is  expected  to  be  much  enhanced 
by  the  launch  of  future  products  such  as  the  X400  with  significant  improvements  in  quality  and 
process  efficiency  throughout  manufacturing  operations.  Customer  satisfaction  levels  are  planned 
to  surpass  leading  German  and  Japanese  competitors. 

This  submission  on  behalf  of  Jaguar  Cars  Limited  identifies  the  Launch  Team  Structure  and 
working  ethic  together  with  8  key  elements,  which  constitute  the  Launch  Processes.  The 
combination  of  these  has  contributed  to  the  quality  of  new  model  launches,  although  the  Cultural 
Changes  within  Jaguar  have  been  without  doubt  the  single  biggest  factor. 
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The  Launch  Team 


Approximately  16  months  from  Job  1  a  Launch  Team  is  formed,  this  comprises  in  the  order  of  200 
full  time  Product,  Manufacturing  and  Production  Engineers,  Purchase  agents,  Finance  Analysts 
and  On-site  Supplier  Representatives.  The  team  is  collocated  at  the  production  site  and  effectively 
eliminates  departmental  barriers,  and  creates  the  sense  of  urgency  necessary  for  rapid  change. 
They  as  a  team  have  responsibility  for  ensuring  the  project  goes  into  production  on  time,  on 
budget  and  most  importantly  achieving  the  required  quality  objectives. 


Structure  and  Managers 

The  structure  of  the  Launch  Team  has  in  three  legs,  where  activities  of  a  similar  nature  are 
grouped  together.  The  car  itself  is  broken  it  to  two  areas.  Body  and  Trim  being  controlled  by  one 
manager  and  Powertrain,  Chassis  and  Electrical  by  a  second.  The  third  manager  of  the  team  will 
control  the  Manufacturing  Process  and  Facilities.  These  managers  are  selected  such  that  their 
characteristics  and  strengths  are  a  good  match  for  those  required  to  fulfil  the  role.  This  selection 
is  critical  to  the  overall  success  of  the  process;  its  importance  is  often  overlooked,  as  there  is  often 
pressure  to  take  the  three  best  managers  from  the  organisation  when  the  project  is  of  a  critical 
nature. 

With  the  Body  and  Trim  the  main  focus  is  on  cosmetic  appearance  and  dimensional  stability.  The 
body  panels  must  be  dimensionally  correct  and  their  surface  finish  is  critical  to  the  showroom 
appearance  of  the  car.  The  aim  is  to  create  the  impression  that  the  car  is  hewn  from  a  single  block 
of  material  and  not  a  collection  of  elements  loosely  hung  together  in  a  haphazard  way.  This 
illusion  is  enhanced  if  the  gaps  and  margins  are  as  small  as  possible  and  above  all  uniform.  The 
interior  is  treated  in  a  similar  way  in  an  attempt  to  create  the  illusion  of  a  single  entity  from  a 
form  perspective  while  the  various  materials  are  used  to  create  soft  feel  zones  and  create  a 
luxurious  atmosphere. 

In  this  role  it  is  attention  to  detail  and  a  continuous  drive  towards  cosmetic  perfection  that  is 
required.  The  problems  are  not  often  complex  but  the  solutions  are  difficult  to  achieve,  but  the 
manager  is  required  to  be  ruthless  in  driving  the  team  forward  as  success  is  directly  proportional 
to  effort 

The  Powertrain,  Chassis  and  Electrical  managers’  role  presents  a  veiy  different  set  of  problems  as 
rather  than  cosmetic  appearance  being  important  it  is  the  function  that  is  complex  and  often  at 
issue.  The  solutions  are  complex  in  nature  and  undesirable  system-to-system  interactions  are  not 
uncommon,  but  can  be  difficult  to  isolate  even  before  the  task  of  establishing  root  cause  and  fixing 
the  problem  begins. 

Here  the  role  is  more  one  of  coaching  people  through  the  problem  resolution  process  step  by  step 
without  getting  frustrated.  In  this  area  there  are  few  quick  fixes.  A  structured  approach  to 
establishing  root  cause  and  verification  is  paramount.  Failure  to  do  so  will  result  in  partial 
containment  and  re-emerge  at  a  later  date  as  an  issue  with  the  customer. 

The  third  manager  controls  the  build  process  and  the  installation  and  commissioning  of  any  new 
facilities.  The  size  of  this  role  depends  as  much  on  the  state  of  the  current  production  facilities  as 
it  does  the  size  of  the  product  change.  In  the  case  of  X400,  where  we  are  installing  a  new  paint 
plant,  new  body  construction  facility  and  a  new  trim  and  final  assembly  line,  the  skills  required  is 
of  a  project  management  nature  dealing  with  large  international  companies  who  are  installing  the 
new  production  facilities.  These  contracts  total  over  400  million  dollars  and  thus  the  day  to  day 
activity  is  controlled  by  the  major  contractor  in  each  case  and  the  role  of  the  Jaguar  Launch 
Manager  is  to  oversee  the  individual  projects  ensuring  they  are  integrated  and  deliver  a  total 
production  system  on  time. 
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Operating  Practices 


Some  practices  that  it  would  be  difficult  to  describe  as  processes  but  never  the  less  are  rigidly 
adhered  to  and  provide  the  foundation  upon  which  many  of  the  launch  processes  are  built.  These 
are  outlined  below  in  limited  detail  to  set  the  overall  scene  and  aid  the  understanding  of  the  main 
launch  processes. 

During  the  launch  there  will  be  a  daily  start  up  meeting,  representation  from  all  the  functional 
areas  is  mandatory  and  it  is  normally  attended  by  the  leaders  from  each  team.  Here  all  immediate 
issues  are  discussed  and  a  consensus  as  to  the  way  forward  is  agreed  this  is  then  communicated 
to  the  rest  of  the  team  and  there  is  no  confusion.  Progress  on  vehicle  build  and  major  facilities 
would  be  reported  together  with  an  outline  of  planned  activities  of  the  day  where  they  may  affect 
other  areas.  The  various  program  metrics  would  be  displayed  at  this  meeting,  but  would  only  be 
discussed  if  there  were  a  significant  update. 

All  issues  or  problems  relating  to  the  program  are  tracked  through  AIMS  (Automated  Issues 
Management  System),  which  is  a  comprehensive  database  that  has  various  automatic  dating 
routines,  allows  teams  to  be  assigned  to  individual  problems  and  is  available  through  the  internal 
network  at  all  the  Jaguar  Sites.  All  new  AIMS  are  presented  at  the  start  up  meeting  and  a 
champion  is  assigned,  it  is  then  tracked  as  part  of  the  Champions  AIMS  open  list  to  closure.  The 
team  assigned  will  work  towards  a  resolution  ensuring  all  relevant  data  is  recorded  within  the 
database,  progress  against  each  open  AIMS  is  reviewed  weekly  with  the  Champion  in  an 
Exceptions  Meeting. 

The  Launch  Managers  have  weekly  exceptions  meetings  with  each  of  their  designated  launch 
champions  where  they  review  the  severity  and  progress  of  every  open  AIMS  assigned  to  that 
champion.  These  ensures  progress  is  being  made  and  if  required  the  Launch  Manager  can  give 
direction,  assign  more  resources,  free  up  road  blocks  or  enlist  the  assistance  of  some  external 
specialists. 

If  the  resolution  of  an  issue  requires  a  change  to  the  product,  the  change  instruction  is  issued  via 
WERS  (World  Engineering  Release  System);  this  constituted  a  formal  engineering  release,  which 
will  trigger  an  order  from  purchase.  The  financial  function  within  the  Launch  Team  advises  on  the 
cost  affordability  of  the  potential  solutions  within  AIMS  and  will  now  formalise  those  costs.  Where 
cost  increase  is  involved,  the  launch  philosophy  is  to  protect  the  customer  and  quality  levels,  but 
to  find  ways  to  make  the  change  affordable. 

The  change  notice,  with  agreed  costs  and  lead-times  is  now  taken  to  the  daily  CMM  (Change 
Management  Meeting)  where  the  change  is  formally  agreed.  This  is  chaired  by  the  Launch 
Managers  and  has  attendance  from  Purchase,  PMC  (Production  Material  Control),  Manufacturing 
and  Finance.  The  details  of  each  change  notice  in  terms  of  cost,  lead-time,  ability  to  resolve  the 
issue  and  manufacturing  feasibility  are  presented  and  if  agreed  the  order  will  be  placed  the  same 
day. 

This  relentless  daily  routine  will  handle  over  12000  AIMS  issues,  2000  engineering  changes  and 
co-ordinate  the  build  of  some  600  prototype  vehicles.  These  are  all  monitored  against  key  metrics 
which  are  reviewed  weekly  and  their  progress  against  previously  agreed  targets  reported  at  the 
formal  launch  reviews. 
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Launch  Process  Elements 
Launch  Metrics 

The  program  is  assessed  against  internal  measures  or  Launch  Metrics  which  allow  program-to- 
program  comparisons  and  some  of  which  can  be  carried  forward  into  production  to  drive  continual 
improvement  initiatives.  Metrics  targets  are  established  by  benchmarking  previous  programs  and 
best-in-class  opposition  and  are  set  for  each  build  phase  and  progress  against  these  targets  is 
reviewed  at  monthly  Launch  Reviews. 

Functional  Images: 

The  status  is  reviewed  as  evaluated  by  the  Vehicle  Office  in  areas  such  as  Ride,  Handling,  Steering 
Feel,  Noise  Vibration  and  Harshness,  Braking  Performance  etc.  against  the  objectives  which  were 
set  at  program  approval. 

DVPSOR: 

This  is  the  Design  Verification  Plan  and  Sign  Off  Process,  where  by  the  Engineering  test  status  for 
each  system  and  the  vehicle  as  a  whole  is  reviewed  against  the  predetermined  sign  off 
requirements. 


This  is  the  normal  plant  quality  audit  process  where  faults  are  recorded  and  depending  on  the 
percentage  of  customers  who  would  notice  the  issue  a  demerit  score  is  assigned.  These  scores 
range  from  9  points  where  greater  than  90%  of  customers  would  notice  to  0. 1  where  1%  of  the 
Customers  would  notice  a  particular  issue. 
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AIMS: 


The  number  of  open  AIMS  is  tracked  to  give  an  overall  view  of  the  work  outstanding  for  the 
Launch  Team.  The  closure  rate,  severity  and  progress  towards  closure  are  also  reviewed  as  this 
gives  a  much  broader  view  of  the  health  of  the  program.  A  high  number  of  low  severity  AIMS  with 
a  high  closure  rate  is  obviously  a  more  healthy  state  than  a  low  number  of  high  severity  AIMS 
which  cannot  be  resolved. 


Windnoise: 
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The  windnoise  of  each  vehicle  is  tested  during  the  road  test  and  given  a  standard  rating  on  a  scale  ofl  to  10.  This  is  the 
normal  ongoing  production  process,  which  is  relatively  good  at  detecting  outright  failures  and  trends  as  the  average 
score  is  reported.  To  establish  the  absolute  level  of  performance  one  car  out  of  each  build  phase  is  taken  to  a  high-speed 
wind  tunnel  to  ensure  repeatable  conditions,  this  measurement  in  sones  is  then  compared  to  the  target,  which  was 
established  by  similar  exercises  on  competitor  products. 
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Water  Leaks: 


All  vehicles  are  subject  to  a  monsoon  type  water  test  using  the  standard  production  test  facility; 
the  number  of  water  leaks  per  car  is  then  tracked  against  the  objective  that  was  set  at  program 
approval. 


This  is  assessed  on  a  hydraulic  shaker  rig  called  a  Hydropulse;  the  number  of  squeaks  and  rattles 
per  car  is  then  reported  against  Program  Objectives.  This  has  recently  been  expanded  to  include 
all  static  noises,  tests  at  extremes  of  temperature  and  50,000  mile  rig  simulations.  These  are 
reviewed  at  the  launch  review,  but  are  reported  separately,  as  there  are  no  program  objectives  or 
historical  data  to  use  as  a  comparison. 


12.7 


Parts  Status: 


Each  part  is  required  to  be  signed  off  against  its  Functional,  Dimensional  and  Cosmetic 
requirements.  This  is  achieved  via  PSW  (Part  Sample  Warrant),  each  supplier  is  required  to  submit 
a  PSW  promise  date  against  the  latest  level  parts.  These  promises  are  then  plotted  against  the 
program  timing  to  establish  what  percentage  of  parts  will  be  approved  at  each  build  phase.  This 
gives  a  measure  of  the  risk  to  production,  as  vehicles  with  non-approved  parts  cannot  be  sold. 
Obviously  the  level  of  ongoing  engineering  change  has  a  direct  impact  on  PSW,  as  a  modified  part 
must  be  reassessed  against  the  new  criteria. 

Manufacturing: 

A  manufacturing  status  against  plan  will  be  reviewed  in  the  areas  of  Facilities,  Process  and 
Training  giving  an  overall  Manufacturing  Risk  Assessment. 
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Launch  Reviews 


These  are  held  monthly  and  are  chaired  jointly  by  the  Executive  Director  of  Manufacturing  and 
the  Vehicle  Line  Director,  its  purpose  is  to  establish  the  overall  health  of  the  launch  which  can  the 
be  communicated  to  other  areas  of  the  company  not  directly  involved. 

The  agenda  is  set  by  the  Launch  Chief  Engineer,  but  as  a  minimum  it  will  include  a  review  of  each 
of  the  Launch  Metrics  against  the  current  build  phase  target  and  the  level  of  confidence  of 
achieving  the  Program  Targets  at  Job  1.  Additional  items  would  be  included  where  a  compromise 
was  being  proposed;  the  original  targets  had  been  changed  due  to  external  factors.  The  senior 
nature  of  the  attendance  also  allows  management  guidance  to  aid  resolution  of  key  issues  and 
concerns,  having  gained  authority  for  a  change  in  this  forum  it  is  rarely  challenged  which 
minimises  the  delays  often  caused  by  interdepartmental  objectives. 

Full  Service  Suppliers 

Here  in  addition  to  the  production  of  components,  a  supplier  with  relevant  expertise  in  a 
particular  area  is  given  Design,  Development  and  Release  Responsibility.  This  type  of  action 
usually  covers  major  car  systems  such  as  the  electrical  vdring  System  or  Interior  trim  systems. 

Quality  deliverables  are  defined  as  an  integral  part  of  the  design  brief  or  target  agreement. 
Competitive  quality  criteria  defining  the  product  in  terms  of  the  best  practice  are  identified  and 
the  supplier  is  responsible  for  design,  development  and  production  of  the  component  or  system 
within  the  target  agreement,  in  terms  of  quality  and  cost. 

Suppliers'  participation  is  planned  from  the  concept  stage  of  the  design  process.  Suppliers  are 
fully  integrated,  and  in  Simultaneous  Engineering  teams  work  closely  with  Jaguar  personnel  on 
design  detail  and  manufacturing  feasibility  to  achieve  quality  products,  delivered  to  the  trackside 
at  the  right  time. 

To  enhance  the  launch  support  process.  Executive  Champions  are  appointed,  16  months  before 
Job  1,  to  review  progress  with  suppliers  of  key  components  or  systems,  identified  as  critical  to  the 
vehicle  launch.  These  take  the  form  of  Supplier  Launch  Reviews  targeting  area  of  risk  within  the 
suppliers’  own  launch  plans,  and  so  requiring  discussion  on  training,  recruitment,  facilities, 
tooling  and  sampling  plans.  Supplier  Technical  Assistance  (STA)  are  responsible  for  supporting  the 
Supplier  Launch  reviews  as  facilitators.  As  they  have  Jaguar  Executive  support  the  tendency  is  for 
the  Supplier  Senior  Management  to  get  involved,  giving  focus  and  much  shortened  decision  loops. 

This  ‘team’  approach  of  supplier/ manufacturer  relationships  ensures  rapid  response  to  change, 
with  reduced  costs  and  more  robust  plans  to  get  the  product  to  market  on  time.  Through  this  the 
Suppliers  are  improving  their  own  processes  to  meet  our  needs  and  therefore  placing  themselves 
in  the  best  position  for  future  business  in  the  industry. 
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Electrical  Prove  out 


Vehicle  Electrical  Systems  are  becoming  increasingly  complicated,  new  technologies  have  been 
introduced  to  meet  demanding  product  specification  and  package  constraints  imposed  by  the  low 
sleek  style  including  multiplexed  wiring  and  fibre  optic  systems  with  low  and  high  speed 
communication  networks.  Although  individual  system  functionality  is  generally  understood  and 
well  defined,  the  system-to- system  interactions  are  a  new  voyage  of  discovery  each  time.  The 
Electrical  Proveout  can  be  conveniently  divided  into  two  areas  of  test  work.  Functional  Testing  and 
Measurement  Testing. 

Functional  Testing: 

This  is  a  detailed  test  regime,  which  requires  that  systematic  test  procedures  are  written 
based  on  the  Full  Functional  Specifications  of  each  electrical  system.  These  are  then  carried 
out  to  ensure  that  all  aspects  of  each  individual  system  is  fully  exercised.  Additionally 
branch  analysis  is  performed  on  all  probable  feed  or  earth  paths  to  identify  possible  sources 
of  interaction  and  these  are  incorporated  into  the  test  regime.  All  anomalies  or  unexpected 
events  with  regard  to  the  whole  vehicle  generate  an  AIMS  which  must  be  resolved  by 
establishing  root  cause  and  implementing  containment  with  follow  up  permanent  corrective 
action. 

Measurement  Testing:  .  . 

These  tests  are  carried  out  on  parameters,  which  have  historically  been  an  issue.  Here  it  is 
the  voltage,  resistance  or  current  that  is  measured  and  compared  to  the  expected  results. 
The  areas  of  general  concern  are  usually  quiescent  current  drain  over  time;  lamp  voltages, 
fuel  consumption  accuracy,  fuel  level  and  damping  strategy  and  voltage  drop  across 
connectors  and  switches.  Again  detailed  test  procedures  are  written  which  allow  accurate 
and  comparable  results  to  be  obtained.  With  this  series  of  tests  there  is  more  emphasis  on 
car  to  car  variability,  but  the  resolution  path  through  AIMS  remains  the  same. 

This  systematic  approach  where  every  system  on  the  car  has  its  function  fully  exercised  and  all 
incidents  reported  and  investigated  is  laborious  as  each  vehicle  will  be  subject  to  over  3000 
individual  tests  and  reports  of  20  pages  are  often  generated,  but  its  value  in  preventing  customer 
incidents  is  well  proven.  The  issue  which  we  have  yet  to  resolve  is  as  the  complexity  and 
interactivity  of  the  vehicle  systems  grow  then  we  need  to  supplement  this  type  of  testing  with 
simulation  analysis  which  is  based  on  actual  performance  rather  that  design  intent. 


Drive  Team  Evaluations 

Vehicles  from  later  build  phases  are  subject  to  rigorous  evaluation.  A  team  of  assessors  use  a 
structured  evaluation  sheet  from  which  they  appraise  each  vehicle  in  a  consistent  way.  The  routes 
used  will  be  the  same  for  all  vehicles  and  the  sample  number  will  be  high  as  one  of  the  objectives 
is  to  get  a  view  of  car  to  car  variability.  The  assessment  team  is  drawn  from  a  number  of  different 
areas;  its  composition  will  be  a  mix  of  trained  evaluators  and  project  team  members. 

The  vehicle  is  split  into  its  Functional  Image  categories;  each  is  assessed  on  a  rating  scale  from  1 
to  10.  The  results  are  analysed  at  regular  Wash  Up  meetings  where  the  results  are  compared  to 
the  program  objectives  and  any  specific  issues  will  have  AIMS  raised  against  them 
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Field  Evaluation  Prototypes 

A  fleet  of  early  build  cars  is  allocated  to  various  Senior  Managers  and  Executives,  which  are  run 
as  normal  day-to-day  cars  in  lieu  their  normal  vehicles.  The  placement  is  spread  to  provide  the 
best  coverage  of  all  vehicle  types  and  driving  styles,  some  in  the  UK  which  allows  good  access  for 
fault  diagnosis,  some  in  Germany  where  the  average  speeds  are  generally  veiy  high  and  some  in 
the  USA  where  they  regularly  travel  for  extended  periods  at  constant  speed. 

These  vehicles  are  used  to  simulate  normal  customer  usage  and  a  concern  report  must  be  raised 
as  soon  as  an  issue  arises.  In  addition  they  must  fill  out  a  weekly  questionnaire  similar  to  the 
Drive  Team  documents  that  are  used  to  gather  more  structured  feedback,  to  compare  against  the 
drive  team  results  and  the  program  objectives. 

The  vehicles  are  returned  to  the  launch  team  workshops  for  1  day  each  week  where 
measurements  will  be  taken  and  normal  maintenance  checks  carried  out  to  assess  if  the  vehicle  is 
performing  as  expected.  This  allows  any  updates  to  be  made  where  revised  production  parts  have 
become  available,  the  users  then  collect  the  vehicles  and  a  feedback  clinic  is  held  where  issues  are 
discussed  to  establish  their  severity  and  the  users  may  be  requested  to  pay  particular  attention  to 
certain  areas. 

All  issues  identified  in  this  process  are  again  resolved  through  AIMS. 

Batch  and  Hold 

Following  the  decision  to  authorise  production  ramp  up  into  saleable  cars,  the  cars  are  held  in 
batches  prior  to  dispatch.  A  NOVA-C  audit  is  completed  on  a  representative  sample  from  each 
batch,  any  serious  issues  found  in  the  audit  are  then  used  to  construct  an  inspect  and  rectify  list. 
The  whole  batch  is  now  assessed  against  this  list  prior  to  being  dispatched. 

Conclusion 

In  summary,  the  way  Jaguar  launches  New  Products  has  become  a  strength  rather  then  the 
weakness  it  once  was,  it  is  an  iterative  process,  which  seeks  to  build  on  the  experience  gained 
from  the  previous  launches  while  continually  attempting  to  eliminate  the  weaknesses.  The  over 
riding  message  should  be  that  success  is  in  the  detail  and  if  the  daily  routine  is  ignored  then  a 
poor  result  will  follow. 

Jaguar  is  no  longer  in  the  survival  mode  of  the  early  1990’s,  but  has  a  sound  business,  poised  for 
profitable  growth  over  the  next  few  years  It  is  competing  for  the  UK  in  very  aggressive  world-wide 
markets  for  luxury  cars,  but  is  very  aware  that  a  single  poor  product  launch  will  undo  the  good 
progress  made  todate. 
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Abstract 

In  the  construction  industry,  as  with  defence 
procurement,  there  is  pressure  from  clients  to  deliver 
‘value’,  i.e.  be  cost  effective  and  shorten  product 
delivery  times,  i.e.  a  ‘lean’  approach.  In  addition 
(especially  in  the  UK),  there  is  a  drive  for 
contractors  to  deliver  a  ‘service’  over  a  specified 
period  of  years  rather  than  just  a  physical  asset  (the 
building). 

It  is  recognised  that  the  key  issue  is  getting  the 
product  right  as  early  as  possible  -  achieving  early 
‘fixity’  of  design  although  this  is  not  always  possible 
as  clients’  needs  may  evolve  and  change  tluoughout 
a  project. 

However,  a  number  of  recent  initiatives  (mostly 
client  inspired)  have  been  shown  to  provide  a  route 
to  increased  value  and  ability  to  manage  clients’ 
evolving  needs. 

Tliese  are: 

1 .  Partnership  agreements  (long  term 
relationships  between  client  and  contractors,  ratlier 
dian  just  on  a  project  basis). 

2.  Multi-disciplinary  project  teams  (client  and 
contractor  sharing  the  decision  making). 

3.  Maximum  standardisation  of  components 
and  systems  (reduction  of  design  errors,  more 
reliable  procurement  supply  chains). 

4.  Structured  approach  to  design  decision 
making  (tools  to  handle  design  dependencies)  and  to 
permit  a  concurrent  design  approach. 

These  ideas,  with  adaptation,  should  equally  be 
applicable  to  defence  procurement. 

Introduction 

It  is  well  known  that  there  have  been  many  ‘new’ 
developments  in  manufactiuing  management 
practices  over  the  last  2  decades  (supply  chain 
management,  just-in-time,  TQM,  lean 
manufacturing,  for  example)  which  have  been 
introduced  to  improve  efficiency  and 


effectiveness  as  a  consequence  of  increasing  global 
competition. 

These  developments  were  primarily  initiated  by  and 
mostly  implemented  in  industries  such  as 
automobiles  and  consumer  electronics  where  there  is 
the  fiercest  competition  between  worldwide  brands 
and  where  products  are  manufactured  on  a  large 
scale. 

More  recently,  the  construction  industry  has  started 
to  take  up  some  of  these  ideas  and  adapt  them  to  an 
obviously  different  ‘manufacturing’  industry 
simation  with  a  view  to  improving  the  efficiency  of 
the  industry.  The  need  for  this  is  the  same  as  for  the 
generality  of  manufacturing  -  increasing  global 
competition  and  consequently  more  demanding 
customers. 

The  evidence,  so  far,  is  that  these  ideas  do  have 
value  in  improving  the  construction  process  but  that 
direct  transfer  from  the  automotive  industry  (which 
is  what  most  people  have  tried  to  do)  is  not  the  right 
way.  A  better  parallel  is  with  aerospace.  Tliis 
aerospace  ‘model’  when  applied  in  the  construction 
industry  (another  project  type  industry  with, 
therefore,  strong  project  management  skills)  can 
provide  a  usefiil  set  of  experiences  which  may  well 
be  of  value  to  defence  procurement  and  project 
management  generally. 

The  similarities  between  the  construction  and 
defence  industries 

Both  of  these  industries  have  many  similarities  and 
both  differ,  significantly,  from  consumer  products 
industries  such  as  automotive: 

•  Customers  are  knowledgeable  and  will  define 
functional  requirements  of  the  product  and  will 
frequently  be  involved  in  developing  the 
technical  specifications,  devising  the  technical 
solutions  and  monitoring  progress. 

•  Products  are  often  ‘unique’  in  that  they  are 
supplied  in  small  volumes  or,  if  in  larger 
volumes,  often  with  many  variants. 

•  The  design  and  development  stage  is  a 
significant  part  of  the  supply  process  both  in 
terms  of  its  importance,  its  complexity  and  its 
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length.  Customer  involvement  in  this  stage  is 
usual. 

•  Individual  units  of  product  are  of  high  value. 

•  Supply  chains  are  frequently  devised  on  a 
‘project’  basis  and  often  are  short  lived. 

Current  trends  in  the  construction  industry 

Traditionally  tlie  industiy  has  operated  with  a 
variety  of,  often  complex,  contractual  relationships. 
The  roles  of  client,  architect,  specialist  consultant, 
quantity  surveyor,  main  contractor  and  sub¬ 
contractors  can  vary  between  projects  and  can 
depend  (in  practice)  on  the  relative  skills, 
involvement  and  commercial  strengths  of  the 
different  parties.  The  allocation  of  risk  between  the 
parties  is  often  not  clear  leading  to  a  tendency  for 
adversarial  relationships.  This  is  not  helped  by  the 
low'  margins  that  contractors  often  work  to  and  the 
financial  resources  needed  for  a  major  construction 
project. 

Two  serious  investigations  into  the  operation  of  the 
construction  industry  “Constructing  the  Team”  (Ref 
1)  and  “Rethinking  Construction”  (Ref  2)  have 
highhghted  the  need  for  improvements  in  tlie 
traditional  approach  to  construction  management 
and,  in  tlie  case  of  the  Egan  report,  set  quantitative 
targets  to  aim  for. 

These  investigations  reflected,  in  part,  tlie  concern 
tliat  major  clients  (major  property  developers,  retail 
and  hotel  chains,  British  Airports  Authority  and 
public  sector  procurement  bodies  in  defence  and 
health  sectors)  were  not  necessarily  getting  best 
value  for  money  from  their  consfruction  projects,  as 
well  as  natural  concern  to  have  an  efficient 
construction  industry  (it  comprises  ~  10%  of  GDP  in 
the  UK). 

Consequently,  the  major  chents,  together  with  some 
of  the  more  far  seeing  contractors,  have  largely  been 
responsible  for  introducing  new  ways  of  managing 
the  construction  process  using  manufacturing 
management  experience  suitably  adapted  to  the 
construction  industry. 

Latterly  also,  there  has  been  a  greater  empliasis  on 
the  whole  life  costs  of  a  construction  reflecting  the 
fact  that  clients  see  a  building  or  other  installation 
not  as  an  entity  in  itself  but  as  a  means  of  delivering 
a  service  over  a  long  period  (accommodation,  a 
working  environment  or  transport  service,  for 
example).  This  is  leading,  in  the  UK,  to  the  Private 
Finance  Initiative  and  similar  mechanisms  where  a 
supplier  is  contracted  to  build,  own  and  deliver  a 
service  over  a  period  of  years. 


In  such  a  situation  there  is  added  need  for  the  client 
to  get  closer  to  the  contractor,  to  develop  a  non- 
adversarial  relationship  and  to  work  together  to  add 
value  to  the  project  as  they  have  a  shared  interest  in 
long  term  success. 

It  is  not  difficult  to  see  the  parallels  between  this  and 
defence  procurement.  Hence,  we  can  anticipate  that 
these  initiatives  in  the  construction  industry  have 
relevance  to  defence  procurement.  Perhaps,  also, 
there  are  new  defence  procurement  ideas  which 
could,  with  advantage,  be  translated  to  construction? 

Examples  of  new  approaches  to  construction 
management 

I  have  taken  4  separate  developments  in  consfruction 
management  to  illustrate  the  nature  of  the  change 
occurring  in  the  construction  industiy. 

These  different  developments  are  not  mutually 
exclusive,  in  fact  they  are  complementary 
developments.  Inevitably,  of  course,  some 
contractors  and  clients  have  emphasised  one  or  more 
of  these  rather  than  others.  It  is  not  sensible  to 
introduce  too  many  changes  in  a  company  culture  all 
at  the  same  time  -  we  have  all  suffered  from 
‘initiative  overload’. 

However,  the  common  theme  of  all  of  them  is 
improving  the  efficiency  and  predictability  of  the 
whole  construction  process. 

1.  Partnership  Agreements  This  is  a  concept 
which  originated  in  another  ‘project’  type  industry  - 
i.e.  oil  and  gas  field  developments  on  UK 
continental  shelf.  In  the  late  1980’s  -  early  1990’s, 
the  fall  in  the  oil  price  led  to  a  need  to  reduce  field 
development  costs.  Partnership  agreements  where 
the  operators  (the  oil  majors)  entered  into 
agreements  with  their  major  suppliers  were  a  key 
factor  in  this  cost  reduction  programme. 

In  return  for  a  ‘guaranteed’  position  as  a  supplier,  a 
major  contractor  agreed  to  a  share  in  the  project  risk 
(and  reward  if  projects  came  in  ahead  of  time  and 
budget).  Clearly  this  implied  a  non-adversarial 
relationship  leading  to  a  more  efficiently  managed 
project  and  lower  project  costs.  This  is  an  advance 
on  the  situation  in,  say,  the  automotive  industry 
where  major  Tier  1  suppliers  have  long  term 
relationslfips  with  the  automobile  companies  and  are 
expected  to  carry  out  and  fund  some  of  the  product 
development  work  on  new  models. 
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Partnership  agreements  on  the  oil  and  gas  industry 
model  are  beginning  to  appear  in  tlie  construction 
industry,  Tliey  are  a  necessary  pre-condition  for 
effective  and  credible  mulU-disciplinaiy'  teams. 

2.  Multi-disciolinarv  Project  Teams  This 
follows  on  from  the  partnership  concept  and  seeks  to 
include  all  relevant  professionals  as  early  as 
possible.  It  is  well  recognised  that  the  cost  and 
difficulty  of  making  a  design  change  increases 
exponentially  as  a  project  progresses  (Ref  3,  for 
example,  although  there  are  many  others). 

Therefore  it  makes  sense  to  prevent  this  as  far  as 
possible.  Co-located  design  teams  which  include 
client,  architect  and  contractors  have  been  found  to 
work.  Fig.  1  shows  how  the  design  decision  making 
in  such  a  situation  compares  with  the  more 
conventional  approach. 

A  further  example  is  the  “Building  down  barriers” 
project  where  the  project  organisation  for  an  MOD 
facility  was  structured  into  core  and  cluster  teams 
This  meets  tlie  dual  need  of  overall,  integrated 
control  of  the  project  together  with  groups 
concentrating  on  more  specific  aspects. 

We  might  anticipate  that  FT  developments  will 
facilitate  this  and  allow  for  effective  teamworking 
without  members  having  to  be  co-located.  This  is  a 
future  challenge  and  opportunity  for  tlie  industry. 

3.  Maximum  Standardisation  and  Pre-ssemblv  of 
Components  and  Systems  This  seems  an  obvious 
route  to  maximising  construction  efficiency  but  it 
has  to  be  done  without  compromising  an  architect’s 
freedom  to  design  innovative  buildings.  A  recent 
study  (Ref  5)  has  explored  this  issue  and  illustrates 
many  case  studies  showing  different  degrees  of 
standardisation  and  the  associated  issue  of  pre¬ 
assembly  off  site. 

Clearly  there  has  to  be  an  element  of  standardisation 
in  any  case,  if  only  at  material  and  component  level 
but  the  issue  is  how  to  optimise  standardisation  at 
the  system  or  building  module  level  but  preserve  the 
arcliitect’s  freedom  to  design  innovative  buildings. 
Standardisation  and  pre-assembly  can  potentially 
lead  to  increased  efficiency,  higher  quality  (factoiy 
ratlier  than  site  manufacture)  and  reduction  of  the 
amount  of  hazardous  site  work.  There  is  a  downside 
to  this  which  is  related  to  the  nature  of  the 
construction  industiy.  For  economic  off  site  pre¬ 
assembly  and  significant  standardisation  of  systems 
and  modules  there  must  be  an  accurately  predicted 
demand  so  that  manufacturers  can  plan  their 
operations.  The  early  experience  of  ‘system 


building’  for  UK  public  sector  construction  (houses, 
schools,  for  example)  was  not  good  in  that  there  was 
little  overall  co-ordination  of  procurement.  We  now 
know  much  more  about  supply  chain  management 
and  business  processes  so  that,  in  principle,  we  can 
achieve  the  economic  potential.  Research  into  the 
relatively  simple  operation  of  constructing  a 
concrete  frame  for  a  building  (Ref  6)  has  shown  that 
attention  to  elementary'  supply  chain  management 
issues  can  reduce  construction  cycle  time  by  28%. 
This  indicates  the  potential  gains  available. 

As  has  been  noted  earlier,  construction  is  a  ‘project 
based’  industiy  with  project  specific  organisation 
and  supply  chains.  However,  the  major  clients 
(utilities,  hotel  and  retail  chains,  public  sector 
bodies)  can  implement  a  more  ‘company’ 
organisation  and  supply  chain  systems  as  they  have 
a  continuous  construction  activity. 

An  example  is  British  Airports  Authority  which  is 
operating  a  policy  of  standard  specifications  and 
preferred  suppliers  of  construction  items.  There  is 
no  evidence  that  this  need  stifle  innovative 
architectural  design  (though  some  architects  might 
dispute  this). 

4.  Structured  approach  to  design  decision  making 
The  client  briefing  and  design  stages  (i.e.  the  pre¬ 
construction  stages)  are  critical  as  errors  and 
omissions  at  this  stage  are  expensive  and  difficult  to 
correct. 

There  has,  therefore,  been  considerable  effort  to 
understand  the  design  decision  making  process  in 
construction,  model  it  and  then  be  better  able  to 
manage  and  monitor  it. 

Modelling  of  the  design  process  analagous  to  the 
product  realisation  models  developed  in  aerospace 
for  example  provides  an  understanding  of  the 
process  at  the  abstract  level  but  does  not  realistically 
map  the  highly  iterative  natiue  of  design.  By  careful 
analysis  of  design  decisions  as  they  are  made  (Ref 
7)  we  can  establish  some  generic  facts  about  design 
decision  making: 

1 .  Irrespective  of  any  other  factors,  the 
process  of  design  decision  making  follows  a 
characteristic  ‘S’  curve  when  plotted  against  time  on 
a  normalised  basis,  although  the  parameters  of  the 
curve  will  vary  between  narrow  limits  as  indicated 
by  upper  and  lower  bounds  (Fig.  2). 

2.  More  than  70%  of  design  decisions  at  the 
system  and  sub-system  level  are  common  although 
the  sequence  can  differ  significantly.  This  sequence 
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difference  can  arise  from  a  number  of  causes;  little 
inter-dependence  between  decisions  at  this  level,  the 
highly  iterative  nature  of  the  design  process  or  tlie 
organisational  structure  (the  relationship  between 
the  participants  in  the  decision  making  process). 

3 .  The  most  common  drivers  for  design 
decisions  are  functionality,  cost  and  interface  issues. 

4.  Design  on  a  ‘day-to-day’  basis  is  a  highly 
iterative  process  not  easily  describable  by  structures 
wliich  have  a  consistent  relationship  to  the 
established  ‘macro’  models  of  the  design  process 
which  are  used  as  high  level  abstractions  to  depict 
the  major  design  output  stages. 

From  tliis  we  can  develop  a  simple  5-step  guide 
(Ref  8)  to  plan  the  design  decision  and  timescale  as 
a  management  /monitoring  tool: 

1 .  Identify  the  time-frame  allowed  for  the 
design  stage  from  initial  client  contact  to  design 
completion. 

2.  List  the  decisions  to  be  made  at  system  and 
sub-system  level  (typically  80-100  decisions  for  a 
building). 

3.  Produce  a  table  of  the  number  of  decisions 
(not  the  actual  decisions)  against  time  in  steps  of 
10%  of  allowable  design  time-frame. 

4.  Order  the  decisions  in  terms  of  the 
available  skills,  resources,  information  and 
constraints  at  the  appropriate  stages  of 
decision/timetable. 

5.  Produce  a  final  table  of  decisions/time¬ 
scale. 

Ordering  the  decisions  is  the  difficult  step  but  there 
is  now  available  a  method  of  analysing  and  ordering 
tlie  design  dependencies  for  this  purpose  using  the 
design-structure-matrix  approach  (Ref  9). 

Conclusions 

This  paper  has  illustrated  the  many  similarities 
between  defence  procurement  and  construction 
projects.  The  competitive  pressures  on  the 
construction  industry  and  their  ever  more  demanding 
clients  have  led  to  significant  adaptation  in  business 
practice. 


Examples  of  four  areas  where  this  has  occurred  (or 
is  being  addressed)  have  been  discussed.  They  have 
been  selected  partly  because  the  ideas  would  seem  to 
be  applicable  also  to  defence  procurement  where 
there  is  a  need  to  improve  business  processes  for 
very  similar  reasons. 

It  has  been  found  that  direct  transfer  of  practices 
across  from  manufacturing  into  construction  is  not 
correct.  Ideas  have  to  be  adapted  and  amalgamated 
with  the  best  practices  in  the  construction  industry, 
e.g.  on  project  management.  The  same  will  occur  if 
the  ideas  are  applied  to  defence  procurement. 
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Fig.  la;  Wlio  made  the  design  decisions  (traditional  project) 


Fig.  lb:  Who  made  the  design  decisions  (team  based  project)  (Ref.  10) 
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Abstract 

The  United  States  Air  Force  Operational 
Test  and  Evaluation  Center  (AFOTEC) 
is  now  conducting  Operational  Impact 
Assessments  (OIA)  of  certain 
Department  of  Defense  (DoD)  Mission 
Areas  in  order  to  inform  the  appropriate 
warfighting  Commanders  in  Chief 
(CINCs)  of  the  mission  impact  of 
operating  a  new  weapon  system  in  the 
field.  AFOTEC  has  taken  this  additional 
responsibility  to  increase  the  value  of 
our  test  reports.  The  evaluation 
methodology  for  conducting  both 
operational  testing  and  OlAs  is  called 
the  Scope/Cost  Process  (SCP).  This 
methodology  is  a  five-step  process:  1) 
System  Introduction,  2)  Understand 
Battlefield  Operations,  3)  Operational 
Risk/Impact  Analysis,  4)  Develop  and 
Cost  OT&E  Options,  5)  Option  Selection 
and  Tasking.  AFOTEC  is  currently  test 
planning  Wideband  Gapfiller  Satellite 
(WGS).  Along  with  an  evaluation  of  the 
system's  operational  effectiveness  and 
suitability  during  the  Multi-service 
Operational  Test  and  Evaluation 
(MOT&E),  AFOTEC  will  conduct  an  OIA 
on  how  WGS  supports  the  operational 
tasks.  This  paper  explores  the  SCP 
process  for  Wideband  Tactical 
Communications. 

Paper 

1.0  Background. 

AFOTEC  has  been  designated 
by  the  DoD  as  the  lead  operational  test 
agency  for  the  Wideband  Gapfiller 


Satellite  (WGS)  System.  As  such, 
AFOTEC  developed  a  strategy  for 
operationally  testing  WGS  and  its 
operational  interfaces.  The 

methodology  used  to  develop  an 
evaluation  strategy  to  test  weapon 
systems  is  called  the  Scope/Cost 
Process  (SCP).  While  it  encompasses 
the  traditional  system-level  initial 
operational  test  and  evaluation  (lOT&E) 
which  determines  system  operational 
effectiveness  and  suitability,  SCP  also 
includes  an  assessment  of  how  a  new 
system  interacts  with  the  entire 
operational  battlespace.  For  WGS,  the 
lOT&E  will  be  a  multi-service  OT&E. 

2.0  Scope/Cost  Process. 

The  SCP  process  consists  of  five 
steps  which  produce  an  evaluation 
strategy  which  culminates  in  a  Tasking 
Order  (TO).  The  designated  test  team 
uses  the  evaluation  strategy  to  develop 
a  test  concept.  The  five  steps  are:  1) 
System  Introduction,  2)  Understand 
Battlefield  Operations,  3)  Operational 
Risk/Impact  Analysis,  4)  Develop  and 
Cost  OT&E  Options,  5)  Option  Selection 
and  Tasking.  Step  1  is  to  gather  design, 
performance,  and  system  employment 
information  about  the  system  under  test. 
In  order  to  carry  out  step  2  (understand 
battlefield  operations),  the  SCP  team 
consults  experts  from  major  commands, 
unified  commands,  and  joint  commands; 
and  studies  military  doctrine.  To  focus 
the  analysis,  a  particular  mission  area  is 
selected.  Step  2  could  extend  a  long 
period  because  of  the  breadth  of 
information  available,  but  is  usually 
limited  by  the  demand  for  a  TO  (Step  5). 
Step  3  is  an  analysis  to  determine  what 
failures  have  the  most  impact  to  the 
mission.  Steps  2  and  3  flow  together  to 
produce  the  OIA  questions  as  depicted 
in  Figure  1 .  AFOTEC  does  not  have  the 
resources  to  answer  all  of  the  OIA 
questions  so  step  4  is  required.  Step  5 
results  in  a  Tasking  Order  (TO)  which  is 
signed  by  the  Vice-Commander.  The 
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TO  directs  the  designated  test  team  to 
initiate  detailed  test  planning.  The  test 
team  will  then  develop  a  test  plan  to 
conduct  the  lOT&E  and  answer  the 
selected  OIA  questions. 


2.1  System  Introduction. 

Once  launched  into 
geosynchronous  orbit,  the  Wideband 
Gapfiller  Satellites  will  become  a  part  of 
the  Wideband  Satellite  Constellation 
which  includes  the  Defense  Satellite 
Communications  System  (DSCS),  Ultra 
High  Frequency  (UHF)  Follow-on  (UFO) 
communication  satellites  and  the  Global 
Broad  Service  (GBS).  The  WGS  will  be 
controlled  by  the  existing  DSCS 
Operations  Center  (DSCSOC).  The 
WGS  will  use  two  frequency  bands;  X 
band  and  Ka  band  (One  way  Ka  for 
GBS  and  2-way  Ka  for  other 
communications). 

The  WGS  primarily  supports 
deployed  tactical  forces  but  must  also 
be  capable  of  supporting  strategic  and 
tactical  nuclear  missions  areas.  WGS 
supports  the  following  DoD  designated 
mission  areas: 

•  Strategic  Offense 

•  Strategic  Defense 

•  Land  Warfare 

•  Air  Warfare 

•  Naval  Warfare 


•  Theater  Nuclear  War 

•  Inter-Theater  Airlift 

•  Inter-Theater  Sealift 

•  Intra-Theater  Airlift 

•  C3I  Programs 

•  Special  Operations  Forces 

•  Strategic  Command,  Control,  and 
Communications  (C3) 

•  Strategic  Command  and  Control 
(C2) 

•  Strategic  Surveillance  and  Warning 

•  Strategic  Communications 

•  Theater  and  Tactical  Programs 

•  Theater  C2 

•  Tactical  Communications 

•  Navigation/Warfare  C2 

•  Land  Warfare  C2 

•  Air  Warfare  C2 

•  Naval  Warfare  C2 

•  Theater  Nuclear  C2 

•  Support  &  Base  Communications 

•  Long  Haul  Communications 

•  Space  Launch  Orbital  Support 

•  Weather  Services 

•  Non-DoD  Communications. 

To  make  the  SCP  process 
manageable,  the  mission  selected  was 
Special  Operations  Forces  in  step  2. 
Any  of  the  above  tactical  missions  could 
have  been  selected,  however,  the 
Special  Operations  Forces  mission  had 
previously  been  expanded  by  another 
AFOTEC  team  (CV-22). 

2.2  Understand  Battlefield 

Operations 

Figure  2  illustrates  the  Joint 
Operational  Requirements  Document 
Special  Operations  Forces  (SOF) 
Mission  Profile  Key  Performance 
Parameters  (KPPs).  The  CV-22  team 
came  up  with  the  following  mission 
breakdown: 

•  National  Goal:  Ensure  Liberty  for  all 
United  States  (US)  Citizens. 
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•  National  Security  Objective:  Defeat 
Aggression. 

•  National  Military  Objective:  Defeat 
Regional  Aggression. 

•  Regional  Objective:  Defeat 

Unconventional  Threats  to  US 
Citizens  and  Interests  Overseas. 

•  Operational  Objective:  Retrieve 

Military  Personnel  in  Distress. 

•  Operational  Task:  Recovery  of 

Isolated  Personnel. 

An  operational  task  is  defined  as  a 
specific  event  on  the  battlefield. 

For  WGS,  the  mission  breakdown 
focused  on  expanding  the  support 
element  of  Wideband  Tactical 

Communications  within  Special 

Operations  Forces.  Wideband  Tactical 
Communications  was  broken  down  into 
the  following  functional  elements: 

•  Planning  (Deliberate/Crisis) 

•  Force  Generation  and  Direction 

•  Situation  Monitoring 

•  Network  Modification 

•  Force  Management. 

Each  function  was  then  defined  as 
either  a  mission  element  or  a  support 
element.  See  Figure  3. 

Additional  information  was 
gathered  on  Wideband  Tactical 
Communications,  but  will  not  be 
discussed  here  due  to  time  and  space 
limitations. 

This  step  establishes  what  is 
necessary  to  accomplish  an  operational 
task.  The  next  step  determines  what  is 
sufficient  to  accomplish  an  operational 
task.  Both  are  required  for  an 
operational  task  to  succeed. 

2.3  Operational  Risk/Impact 

Analysis. 

The  next  step  is  to  analyze  the 
operational  risk  or  impact  to  adding  the 
system  under  test  (WGS)  to  the 
battlefield.  This  analysis  is  a  four  step 
process  which  looks  at  each  element  in 
the  mission  decomposition  (expansion): 


•  Element  impacts  on  operational  task 
(mission). 

•  Element  impacts  on  other  elements. 

•  Element(s)  impacts  on  the  system 
(WGS). 

•  System  impacts  on  the  element(s). 
See  Figure  4.  Next,  a  Cause  Effect 
analysis  was  accomplished.  Cause 
Effect  is  done  by  looking  at  the 
operational  task  (Recover  Isolated 
Personnel)  and  work  backward 
determining  ail  the  possible  failures  that 
could  occur  which  would  prevent  the 
accomplishment  of  the  operational  task. 
This  analysis  helps  the  team  develop  an 
evaluation  strategy  (see  Figure  1)  by 
focusing  on  what  areas  have  the  most 
impact  on  mission  accomplishment. 
These  focus  areas  become  the 
evaluation  areas  for  the  OIA.  From  the 
evaluation  areas  come  the  evaluation 
objectives.  Next,  questions  were 
developed  which  the  test  team  could 
answer  during  test  execution.  The 
answers  to  selected  OIA  questions  (as 
determined  by  the  final  two  steps)  will 
be  documented  in  the  WGS  MOT&E 
Report. 

For  WGS,  the  Evaluation 
Strategy  is  the  following: 

•  Use  the  SCP  to  ensure  test  program 
is  cost  effective  and  provides 
relevant  information  to  the 
Combatant  Commands. 

•  Maximize  exercise  participation 

within  the  constraints  of  time  and 
resources. 

•  Maximize  data  collection  from 

combined  testing  with  the 
developmental  test  (DT)  community. 

•  Minimize  time  required  for  the 
dedicated  MOT&E. 

•  Eliminate  AFOTEC  testing  of 

specification  level  requirements 
(Ensure  the  DT  community  will 
cover). 
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ops  Risk  Impact  Assessment 


1.  Element  Impacts  on  Operational  Task  (mission) 

2.  Element  impacts  on  other  elements 

3.  Element(s)  Impacts  on  system  (System  dependence  on  Element(s)) 

4.  System  impacts  on  the  Element(s)  _ 


1  Destroy  Bridge 

2  Destroy  Dock  Yard 

3  Destroy  Aircraft 


Support  Elements 


Figure  4.  Operational  Risk/Impact 
Assessment 


Wideband  Tactical 
Communications 

•Is  there  adequate  comm? 

•Is  comm  available  to  support  dissemination  of  ATO’s? 

•Can  comm  handle  the  load  and  time  restraints  (WX, 

Intel,  Mission  Planning...) 

•Is  secure  comm  available? 

•Deter/counter  Info  Warfare  attacks? 

•How  long  to  set  up  to  initially  support  theater  operations? 
•Allocation  of  comm  assets? 

•Is  the  comm  timely  and  sufficient? 

•Is  the  jamming  protection  adequate  in  an  hostile  environment? 

•Is  the  jamming  protection  adequate  in  environment  of  “friendly” 
jamming? 

•Are  the  tactical  terminals  easily  transportable  in  the  theatre? 

•Are  the  tactical  terminals  ruggedized  for  the  theatre? 

•Can  the  tactical  networks  be  efficiently  and  effectively  configured 
real  time  in  theatre? 


Figure  5.  Sample  of  OIA  Questions  for 
WGS 
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The  Evaluation  Area  is:  Timely 
communications  (via  satellite)  to  tactical 
units  in  the  field. 

The  Evaluation  Objectives  are: 

•  Communication 
Planning/Managment 

•  Communication  Performance 

•  Communication  Access 

•  Susceptibility. 

Some  of  the  OIA  questions  are  listed  in 
Figure  5. 

2.4  Develop  and  Cost  OT&E  Options. 

The  next  step  is  to  develop  and 
estimate  costs  for  different  test  options. 
The  AFOTEC  Director  of  Operations  will 
review  and  determine  the  best 
alternative  considering  the  resources 
(manpower,  money,  time)  available. 
AFOTEC  has  a  priority  list  of  programs 
based  on  Major  Commands  priorities, 
the  size  of  the  program,  and  the  value  of 
operational  test  information  to  the 
CINCs. 

The  Director  of  Operations  also 
determines  what  products  the  test  team 
will  produce.  For  WGS,  the  test  team 
will  produce: 

•  Early  Operational  Assessment 
(EOA).  Both  program  information 
and  early  OIA  information  will  be 
included.  This  is  due  to  the 
Milestone  Decision  Authority  by  Aug 
00. 

•  Operational  Assessment  (OA). 
Additional  OIA  information  and  the 
program  issues  affecting  testing  will 
be  included.  Due  to  the  DoD 
Director,  Operational  Test  and 
Evaluation  (DOT&E)  after  the  WGS 
Critical  Design  Review. 

•  MOT&E  Report.  Will  contain  the 
OIA  results  plus  answer  whether  the 
WGS  system  is  effective  and 
suitable.  Due  to  the  Major 
Commands  before  the  WGS  Initial 
Operating  Capability  (IOC)  decision. 

2.5  Option  Selection  and  Tasking 


The  final  step  of  the  initial  SCP 
process  is  option  selection  by  the 
Director  of  Operations.  This  is 
documented  by  a  Tasking  Order  which 
is  signed  by  the  AFOTEC  Vice 
Commander. 

3.0  Summary. 

The  SCP  process  is  a 
methodology  to  ensure  AFOTEC  gives 
the  Combatant  Commands  relevant 
information.  This  information  is  given 
before  a  new  system  is  deployed,  thus 
allowing  the  CINCs  to  decide  how  to  use 
a  system  for  maximimum  effectiveness 
and  suitabilty. 
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Abstract 

Photography’s  inestimable  contribution  to 
science  and  technology  is  beyond  question. 
Applied  photography  is  that  area  of  work 
which  assumes  an  integral  part  of  a  research 
project  and  where  the  results  of  photographic 
analysis  form  a  major  source  of  information. 

In  military  research  and  development  or 
testing  there  are  various  fast  and  dangerous 
events  that  need  to  be  recorded  and 
analysed.  High-speed  cameras  allow  the 
capture  of  movement  too  fast  to  be 
recognised  by  the  human  eye,  and  provide 
data  that  is  essential  for  the  analysis  and 
evaluation  of  such  events.  High-speed 
photography  is  often  the  only  type  of 
instrumentation  that  can  be  used  to  record 
the  parameters  demanded  by  our  customers. 
Examples  are  shown  and  discussed  where 
this  applied  cinematography  is  used  not  only 
to  provide  a  visual  record  of  events,  but  also 
as  an  essential  measurement  tool. 

Whereas  traditionally  silver-based 
photographic  films  were  the  only  recording 
means  available,  nowadays  a  variety  of  other 
sensors  are  used.  This  paper  shows  actual 
film  records  from  a  variety  of  weapon  tests, 
highlights  the  increasing  use  of  electronic 
recording  media,  and  introduces  the  strategy 
of  completely  “digitising”  optical  recording  on 
our  ranges  in  the  near  future.  Although  many 
of  the  techniques  discussed  have  been 
developed  specifically  for  armament  testing 
many  of  them  are  equally  applicable  to  other 
industries. 


Introduction 

Almost  from  the  birth  of  photography  film  has 
been  used  as  the  recording  medium  at  our 
test  sites  during  research  and  testing 
experiments.  Colonel  Shrapnel  was 
conducting  his  studies  into  fragmentation  at 
our  site  at  Shoeburyness  in  the  first  half  of 


the  19*'’  century  -  and  almost  certainly  would 
have  used  photography  to  record  the  results 
if  it  had  been  invented  then.  Shoeburyness 
has  been  a  military  test  site  since  1849, 
which  is  about  the  same  time  that 
photography  as  we  know  it  was  discovered. 

Shoeburyness  is  one  of  our  four  Land 
Environment  Ranges.  We  have  five  Air 
Ranges  and  12  sea  Ranges  (although  some 
of  these  are  very  small  facilities).  These 
divisions  of  Land,  Sea  and  Air  are  likely  to 
disappear  in  the  near  future. 

I  intend  to  show  examples  of  how 
photography  has  been  used  as  an  essential 
data-gathering  tool  over  the  years,  and  why 
we  are  now  moving  away  from  film  based 
systems  to  electronic  digital  imagery.  A 
colleague  of  mine  gave  a  paper  on  the 
history  of  optical  trackers  at  the  last  Test 
&  Evaluation  Forum  two  years  ago,  so  I  do 
not  intend  to  repeat  any  of  that  subject. 
Trackers,  and  their  associated  photography, 
are  largely  confined  to  the  Air  Ranges.  My 
examples  are  largely  drawn  from  the  Land 
Ranges  Environment  and  in  particular  from 
our  site  at  Pendine,  where  I  was  Operations 
Manager  for  four  years.  Pendine  is  the 
centre  for  our  test  track  work  and  high-speed 
cine  photography  is  of  particular  importance 
in  that  application.  However  I  must  stress 
that  the  techniques  discussed  can,  and  are, 
universally  deployed  at  any  of  our  range 
sites. 


Background 

With  its  unique  capability  for  expanding  time, 
“freezing”  motion,  and  accurately  recording 
distances  and  angles,  high-speed 
photography  in  all  its  many  variants  has 
established  itself  as  an  extremely  valuable 
measurement  tool  in  the  testing  and 
development  of  weapons  and  armaments. 
Pendine  has  existed  as  one  of  our  prime 
testing  sites  since  1940  where  the  11km 
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beach  forms  an  integral  part  of  the  testing 
facilities.  This  beach  has  of  course  been 
famous  since  the  1920’s  as  the  site  for  many 
attempts  on  the  land  speed  record,  with 
Malcolm  Campbell,  Parry-Thomas  and 
Foresti  all  pushing  the  record  faster  and 
faster  in  the  hectic  period  between  1924  - 
1930.  I  will  return  to  this  topic  later.  The 
1500m  Long  Test  Track  was  built  there  in 
1956  during  the  heyday  of  the  UK  guided 
weapon  development.  Our  main  customers 
have  always  been  largely  from  MOD  and  the 
defence  industries,  but  we  have  on 
occasions,  and  increasingly  nowadays,  done 
work  for  purely  commercial  customers.  For 
example  here  the  test  track  is  being  used  for 
investigation  of  side  wind  loading  on  a  scale 
model  of  British  Rail's  ill-fated  Advanced 
Passenger  Train.  The  great  variety  of 
comprehensive  testing  facilities  that  exist  on 
our  sites  cover  work  ranging  from  arena  trials 
of  fragmenting  warheads,  standard  fuel  fire 
hearth  tests,  shaped  charge  investigation 
work  which  includes  a  lot  of  flash 
radiography  instrumentation,  to  small  arms 
development,  testing  and  proofing.  However 
by  far  the  greatest  demand  for  high-speed 
photography  is  generated  by  trials  using  one 
of  the  three  test  tracks.  We  have  a  short 
200m  track  largely  used  for  dynamic  proofing 
of  warheads;  a  400m  high  G  impact  test 
track  originally  built  for  testing  nuclear 
transportation  flasks;  but  our  main  and  more 
challenging  workload  is  photography  of  trials 
on  the  unique  (to  the  UK)  1500m  long  test 
track.  Since  its  original  usage  the  track 
facilities  have  been  progressively  expanded 
to  cater  for  a  wide  variety  of  different  trials 
which  include: 

•  Terminal  ballistic  trials  and  proof  firings 
of  warheads  against  a  range  of  aircraft, 
missile  and  plate  targets. 

•  Testing  of  proximity  fuzes  against 
targets  at  operational  velocity. 

•  The  study  of  flight  characteristics  of 
aircraft  ejection  seats  and  instrumented 
dummy  behaviour. 

•  The  study  of  submunition  ejection  and 
separation  from  airborne  dispensers. 

•  The  assessment  of  the  performance  of 
aircraft  missile  warning  systems. 

•  Rain  erosion  trials  of  radomes, 
nosecones  and  other  such  materials. 

•  The  study  of  wire  dispensing  systems 
and  high-speed  parachutes.  This 
includes  the  use  of  on-board  sled 
cameras. 

•  The  study  of  the  ejection  and  flight  of  IR 
flares  and  chaff  decoys. 


•  And  latterly  the  work  has  concentrated 
on  the  penetration  performance  of 
aircraft  bombs  and  missiles  against 
representative  concrete  targets. 

In  all  this  type  of  work  the  first  and  foremost 
reason  for  using  high-speed  photography  is 
to  allow  design  engineers  and  scientists  to 
“see"  these  fast,  transient,  often  dangerous 
and  unpredictable  events,  in  order  to  simply 
assess  behaviour  and  make  informed 
qualitative  judgements.  These  are  often  all 
that  is  required  in  order  to  progress  to  the 
next  stage  of  the  experiment.  High-speed 
photography  not  only  provides  these  quick 
look  results  but  also  gives  a  permanent 
record  for  subsequent  detailed  analysis  and 
measurement.  One  advantage  of  the  move 
to  digital  techniques  is  that  it  will  allow  much 
quicker,  almost  instant,  access  to  these 
results  without  waiting  for  films  to  be 
processed.  It  will  also  allow  for  basic 
computational  analysis  actually  on  the  firing 
site. 


Examples 

1  will  now  show  a  series  of  results  using 
actual  film  clips  from  various  trials  and 
explain  what  the  high-speed  cameras  were 
deployed  to  measure. 

Three  or  four  years  ago  we  were  asked  to 
record  the  performance  of  a  90mm  dense 
metal  penetrator  against  reinforced  concrete 
targets.  The  critical  factor  here  was  the 
impact  angle  was  not  to  exceed  the  design 
parameter  of  2  degrees.  The  store  was 
released  into  free  flight  at  the  end  of  the  test 
track  by  an  explosive  bolt  technique,  which 
meant  that  the  impact  angle  could  not  be 
tightly  controlled,  so  this  had  to  be  accurately 
recorded  for  each  of  the  fifteen  firings. 

This  camera  shows  the  pitch  angle  and  also 
allows  impact  velocity  to  be  measured  by  the 
inclusion  of  a  calibrated  scale,  which  is 
surveyed  in  horizontally  on  the  flight  line.  The 
camera  was  running  at  12000pps  and  the 
impact  velocity  was  measured  off  the  films  of 
the  various  firings  as  between  375  and  450 
metres  per  second  (m/s).  On  two  firings  the 
recorded  pitch  angle  exceeded  the  criteria  of 

2  degrees,  one  being  measured  as  6 
degrees  nose  down,  the  other  being  3.5 
degrees  nose  up.  These  firings  had  to  be 
repeated. 
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This  camera  recorded  the  yaw  angle,  again 
running  at  12000pps,  by  looking  vertically 
down  upon  the  event  from  a  10m  high 
gantry.  On  all  firings  the  yaw  angle  never 
exceeded  the  required  2  degrees. 

Another  important  parameter  that  the 
customer  required  us  to  record  was  total 
penetration  time  through  the  target.  This 
could  have  been  achieved  by  timing  the 
rupture  of  two  foil  switches  electronically,  but 
as  they  also  desired  to  see  target  behaviour, 
and  record  exit  angle  and  velocity,  we  used 
high-speed  photography  to  record  all  desired 
parameters.  This  camera  ran  at  lOOOOpps 
and  by  utilising  the  1kHz  calibrated  time 
base  routinely  recorded  on  the  edge  of  every 
film  penetration  time  intervals  ranging 
between  7  to  10.5ms  were  measured.  In  total 
we  had  eight  cameras  viewing  different 
aspects  on  all  fifteen  firings.  Frame  rates 
varied  between  6000  and  15000pps 
depending  on  the  area  under  study.  Besides 
angular  and  velocity  measurements  tumble 
and  spin  rates  were  also  calculated  from  the 
film  records. 

A  similar  trial  but  on  a  much  larger  scale  was 
carried  out  recently  on  the  long  test  track  to 
study  the  penetration  properties  of  various 
configurations  of  aircraft  bombs.  On  these 
firings  many  of  the  requested  measurements 
were  similar  to  those  just  mentioned,  such  as 
pitch  and  yaw  angles  at  impact,  as  well  as 
flight  behaviour  and  velocities.  Again  high¬ 
speed  cine  cameras  not  only  allowed  the 
recording  of  what  happened,  as  all  evidence 
is  of  course  destroyed  by  the  very  nature  of 
the  experiment,  but  also  provided  the  basic 
data  from  which  these  measurements  could 
be  calculated.  In  all  11  cameras  were 
deployed  looking  at  different  aspects  of  the 
impact  and  target  reaction. 

This  camera  recorded  general  flight 
behaviour  at  impact  by  running  at  lOOOpps, 
whilst  this  one  running  at  3000pps  gave  a 
closer  view  of  impact  from  90  degrees  to  the 
line  of  fire  (LoF)  for  pitch  angle  and  impact 
velocity  measurements.  This  one  again  on  a 
10m  gantry  recorded  yaw  angle  and  a 
second  check  of  velocity. 

One  particular  request  on  these  firings  was 
to  record  a  deceleration  profile  of  the  bomb 
during  penetration  of  the  target.  By  running  a 
camera  at  20000pps  we  hoped  to  provide 
enough  data  points  for  this  to  be  analysed 
and  plotted.  On  the  first  firing  it  was 
immediately  apparent  that  debris  obscured 


the  bomb  during  much  of  the  penetration 
phase.  So  we  added  this  carbon  fibre 
extension  piece  to  the  tail  of  the  bomb  in 
order  to  prolong  the  time  it  was  visible  and 
hence  increase  the  data  recording  time.  This 
was  moderately  successful  but  even  running 
at  20000pps  was  not  providing  enough  data 
points  for  accurate  computation,  especially  at 
the  higher  velocities. 

By  using  streak  photography,  that  is  utilising 
the  fact  that  in  a  high-speed  camera  the  film 
is  in  continuous  motion  past  the  gate  rather 
than  being  stationary  at  the  time  of  exposure 
as  in  conventional  motion  picture  cameras, 
we  produced  this  record  of  the  bomb  impact. 
This  is  basically  a  distance  versus  time 
graph  drawn  directly  on  film  thus  allowing 
subsequent  analysis  and  graphical 
presentation  if  necessary.  The  datums  are 
produced  by  fixing  on  to  the  bomb  body  two 
small  reflecting  spheres  and  illuminating 
them  with  conventional  photographic  lighting 
during  the  impact  phase;  thus  effectively 
producing  point  source  datums.  Illuminating 
spots  of  highly  reflective  material  accurately 
placed  in  the  field  of  view  (FOV)  similarly 
produce  scales. 

The  classical  use  of  streak  techniques  is  in 
the  accurate  timing  of  short  duration  self- 
luminous  or  high  contrast  events.  The 
accuracy  and  certainty  of  recording  these 
events  is  of  course  due  to  the  continuous 
recording  ability  due  to  the  lack  of  any  dead 
time  between  subsequent  exposures 
inherent  in  any  framing  camera  regardless  of 
its  frame  rate.  This  slide  shows  the  operation 
of  two  exploding  bridge  wire  (EBW) 
detonators  in  a  new  design  of  electronic 
fuze.  The  time  interval  between  the  two 
functions  was  measured  directly  off  this  film 
as  300  microseconds.  The  fact  that  the  fuze 
was  travelling  down  the  test  track  at  some 
700m/s  made  the  synchronisation  of  the 
event  to  the  camera  running  time  somewhat 
difficult.  However  the  film  does  not  have  to 
be  run  very  fast  through  the  camera  with  this 
technique,  even  to  record  microsecond 
timings,  thus  giving  a  comfortably  long 
recording  time.  I  am  talking  here  about  using 
film  cameras  with  the  optical  block  removed 
for  this  type  of  application.  Time  resolutions 
in  the  nano  and  pico  second  range,  even  out 
to  femtosecond  resolution,  demand  the  use 
of  very  specialised  electronic  image  tube 
cameras  manufactured  for  this  particular 
purpose  and  thus  very  expensive.  The 
streaking  in  the  background  of  this  example 
is  of  course  caused  by  the  forward  motion  of 
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the  carrier  vehicle  and  can  be  used  to 
measure  velocity  by  calculating  the  angle  of 
the  FOV  and  the  time  taken  to  cross  it.  It 
makes  such  measurements  much  easier  if  a 
distance  scale  is  introduced  into  the  FOV  so 
allowing  a  time  versus  distance  computation 
of  velocity  to  be  made  directly  off  film. 

This  slide  shows  a  static  shaped  charge 
explosion  with  such  a  calibrated  distance 
marker  allowing  a  jet  tip  velocity  in  excess  of 
10000m/s  to  be  measured. 

Another  often  used  application  of  streak 
photography  is  in  post-impact  delay  work, 
where  it  is  necessary  to  record  where  and 
when  a  shell  explodes  behind  a  target.  This 
slide  shows  such  an  event  in  a  multi-plate 
target  where  the  edges  of  each  plate  have 
been  enhanced  with  reflective  material  and 
illuminated  to  produce  the  scale. 

Steak  photography  is  of  course  non-pictorial. 
By  using  a  similar  de-blocked  camera  but 
this  time  running  the  film  parallel  to  the  event 
plane,  we  employ  what  has  become  to  be 
known  as  ballistic-synchro  photography.  This 
produces  a  highly  detailed  single  shot  still 
photograph  of  fast  moving  objects.  This 
technique  has  been  historically  used  for 
many  years  to  record  bullets,  shells  and 
projectiles  in  flight  in  order  to  image  such 
details  as  rifling  marks,  driving  bands,  sabot 
separation,  etc.  It  is  often  accompanied  by  a 
somewhat  bastardised  schlieren  technique 
for  visualising  shock  waves.  This  slide  shows 
how  we  met  one  particular  requirement 
which  was  to  show  the  effect  of  ejecting  sub¬ 
munitions  into  a  supersonic  airflow  at  a 
velocity  of  Mach  2.5  (850m/s). 

This  technique  is  not  really  used  for 
measurement  purposes  but  more  for  imaging 
detail  with  great  clarity  and  resolution. 
However  aerodynamacists  can  calculate 
velocities  from  the  angle  of  the  shock  wave. 
If  three  cameras  are  deployed  in  surveyed 
positions  along  the  trajectory  and  the 
projectile  suitably  marked  then  spin  rates  can 
be  determined.  A  similar  technique  is  used 
for  horse  race  and  athletic  photo-finish 
cameras  where  continuous  recording  is 
important  -  but  these  subjects  are  moving  a 
little  slower  than  bullets! 

We  have  been  involved  in  a  large 
programme  of  research  work  over  the  last  8 
to  10  years  or  so  investigating  the  effect  of 
small  fragments  on  aircraft  structures.  This 
was  concerned  both  with  fragmenting  missile 


warhead  design  and  the  effect  of  terrorist 
bombs  on  board  civil  aircraft.  Early  in  this 
programme  it  soon  became  evident  that 
conventional  recording  techniques  would  be 
inadequate  to  provide  the  detailed  records 
required.  This  was  particularly  so  with 
recording  the  impact  angle  of  small  6mm 
cubes  with  velocities  in  excess  of  2000m/s. 
We  were  soon  made  aware  of  the  very  real 
problems  of  attempting  to  record  very  small 
high  velocity  subjects  and  the  limitations  of 
conventional  high-speed  cine  techniques.  To 
overcome  these  problems  we  initially 
illuminated  the  cubes  in  flight  with  a  pulsed 
copper  vapour  laser  to  give  very  short 
exposure  times  in  the  order  of  20 
nanoseconds  (ns).  Even  filming  at  20000pps 
the  magnification  required  only  gave  us  three 
frames  of  usable  information.  One  particular 
advantage  of  laser  illumination,  besides  its 
short  duration  and  high  light  levels,  is  its 
monochromaticity.  By  using  narrow  cut  filters 
all  ambient  light  can  be  eliminated  including 
the  kinetic  energy  flash.  This  slide  shows  the 
clearer  image  thus  obtained. 

These  limitations  very  graphically  showed 
the  shortcomings  of  conventional  high-speed 
photography  in  such  situations,  and  led  us  to 
consider  the  then  new  methods  of  electronic 
imaging. 

By  using  an  image  converter  tube  camera 
running  at  the  equivalent  of  1  million  pictures 
per  second  with  200ns  exposure  times  and 
3ns  between  frames  these  pictures  were 
obtained,  allowing  not  only  impact  angles  to 
be  measured  but  also  spin  rate. 

It  became  necessary  during  the  progress  of 
this  prolonged  research,  which  is  still 
ongoing,  to  ascertain  with  even  greater 
accuracy  the  impact  angle:  as  it  was 
discovered  that  this  had  a  significant  effect 
on  subsequent  target  damage.  At  the  same 
time  the  electronic  imagery  capabilities  were 
moving  on  apace.  These  pictures  were  taken 
with  an  electronic  digital  imagery  camera 
specifically  designed  for  ballistic  range  work, 
the  development  of  which  we  had  helped 
with.  They  allowed  us  to  measure  the  impact 
angle  in  detail  but  also  to  look  at  behind 
target  effects;  and  with  a  double  exposure 
technique  to  measure  impact  and  fragment 
velocities. 

The  use  of  these  electronic  imaging  cameras 
has  now  become  extensive  in  our  type  of 
work.  This  slide  shows  there  ability  to  show 
rifling  details  on  a  30mm  round;  and  this  one 
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shows  the  pitch  and  yaw  angles  of  a  20mm 
round  simultaneously  by  using  a  45  degree 
angled  mirror  in  the  FOV  above  the  flight 
line.  These  cameras  advantages  are  now 
invaluable  during  such  trials  as  these  which 
need  many  firings  to  replicate  unpredictable 
and  spurious  events.  The  use  of  such 
cameras  eliminates  the  exposure  and 
processing  of  many  hundreds  of  feet  of  film; 
and  there  instantly  available  record,  with  on 
site  basic  computer  analysis,  means  that 
they  have  become  very  useful  measurement 
tools.  We  have  six  within  ranges  sector.  A 
colour  version  is  now  available.  Although 
these  particular  cameras  produce  only  one 
still  picture,  albeit  with  multi-exposure 
facilities  if  needed,  the  same  technology  is 
now  being  applied  to  the  development  of 
framing  cameras.  This  is  what  drove  the 
strategic  decision  to  go  totally  digital  and 
over  the  next  two  or  three  years  to  replace  all 
our  film  cameras  with  digital  high  frame  rate 
video  cameras.  Because  the  technology  is 
aligned  to  the  consumer  driven  amateur 
digital  stills  market  it  advances  at  a 
remarkable  pace.  Ultimate  resolution  does 
not  yet  quite  match  that  of  film,  but  I  have 
every  confidence  that  it  will  in  the  very  near 
future.  This  meant  that  we  can  dispense  with 
all  our  film  processing  facilities,  with  the 
consequent  environmental  and  cost  saving 
advantages.  We  will  then  be  able  to  offer  our 
customers  a  more  versatile,  speedier,  state 
of  the  art  solution  to  their  imaging  and 
analysis  requirements;  so  making  the 
decision  to  go  digital  both  inevitable  and  of 
sound  business  sense. 
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155min  FSAPDS  Ammunition  -  Film  Image 


155mm  FSAPDS  Ammunition  -  Electronic  Image 
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Abstract 

DERA  T&E  Ranges  has,  amongst  other  things, 
responsibility  for  measurement  and  control  of  the 
underwater  signatures  of  warships.  Recent 
developments  have  extended  the  capability  of 
degaussing  ranges  from  measurement  of  magnetic 
fields  alone  to  electric  and  pressure  with 
confirmatory  checking  of  acoustic  signatures  too. 
The  electric  field  measurement  is  by  a  novel  method 
using  modelling  from  the  electric  potential  which 
permits  far  easier  deployments  of  the  transportable 
range  than  would  be  possible  if  the  field  itself  was 
measured.  One  consequence  of  the  electric  field  of 
certain  vessels  is  the  magnetic  field  from  the  electric 
current  which  results  from  the  presence  of  an 
electric  field  in  seawater.  Since  the  electric  fields 
from  vessels  are  a  result  of  corrosion,  the  magnetic 
field  from  this  cause  is  known  as  the  corrosion 
related  magnetic  (CRM)  field.  The  CRM  field  is 
significant  as  it  decays  more  slowiy  with  distance 
than  the  magnetic  field  from  magnetic  sources. 
Therefore,  it  offers  the  possibility  of  a  greater  range 
of  airborne  detection.  The  new  multi-influence 
ranges  provide  prediction  of  the  airborne  magnetic 
field  from  both  electric  and  magnetic  sources.  The 
range  design  allows  the  fixed  and  transportable 
ranges  to  be  functionally  Identical  so  measurements 
in  various  parts  of  the  world  can  be  compared 
directly.  The  ease  of  deployment  of  the 

transportable  range  means  it  is  simpler  to  take  the 
range  to  the  warships  than  bring  the  ships  to  the 
range  so  international  collaboration  is  simplified. 
Within  a  month  of  acceptance  of  the  system  vessels 
of  the  Royal  Netherlands  and  Belgian  Navies  had 
been  ranged  in  addition  to  British  Royal  Navy 
vessels. 

The  range  systems  are  described  together  with  the 
algorithms  and  software  and  their  empirical 
validation. 

Introduction 

Ships  and  submarines  may  be  detected  in  several 
ways  by  the  electromagnetic  fields  they  emit.  This 
paper  is  mainly  concerned  with  in-water  fields  and 
their  detection.  Because  of  the  rapidly  increasing 
attenuation  of  electromagnetic  radiation  with 
increasing  frequency,  we  are  mainly  concerned  with 
frequencies  below  3kHz.  Of  course,  in  air,  vessels 
on  the  surface  emit  frequencies  much  higher  than 
3kHz  (for  example  communications  radio  and  radar) 
but  these  travel  only  a  very  short  distance  in 


seawater  so  are  not  a  means  of  underwater 
detection.  Submerged  submarines  do  not  emit  radio 
and  radar  signals  (except  when  an  antenna  is 
deployed  on  the  surface  deliberately)  so  only  low 
frequencies  are  of  interest.  The  measurement  of  the 
underwater  electromagnetic  emissions  from  vessels 
is  necessary  to  ascertain  whether  faulty  equipment 
(which  could  be  fixed)  is  causing  excessive  levels,  to 
evaluate  the  threat  from  mines  and  other  detection 
devices  and  to  enable  the  setting  up  of  any 
countermeasures  which  may  be  used. 

Some  Definitions 

Electromagnetic  ranging  has  been  a  development  of 
degaussing  ranging,  so  the  nomenclature  used  to 
describe  the  various  fields  and  bandwidths  has 
developed  from  that  used  in  the  degaussing 
community. 

The  term  degaussing  applies  to  methods  of  reducing 
vessels’  magnetic  fields  by  applying  electric  currents 
to  degaussing  coils  fitted  within  the  vessel  to 
generate  a  magnetic  signature  (the  magnetic  field 
below  the  vessel)  which  opposes  the  magnetic 
signature  of  the  ferromagnetic  material  in  the  vessel. 
Some  vessels  become  permanently  magnetised 
such  that  demagnetisation  is  necessary.  This  is 
generally  known  as  magnetic  treatment  and  requires 
the  application  of  magnetic  fields  externally  to  the 
vessel  by  means  of  large  coils  (several  hundred 
metres  on  each  side)  on  the  seabed,  large 
structures  which  are  big  enough  to  enclose  the 
entire  vessel  bearing  coils  or  closely  wrapping 
substantial  coils  around  the  vessel. 

The  signature  affected  by  magnetic  treatment  and 
degaussing  is  known  as  the  static  magnetic  field 
(since  it  does  not  move  with  respect  to  the  vessel). 
However,  since  the  vessel  moves  with  respect  to  the 
seabed  (where  a  range  or  a  magnetic  mine  sensor 
might  be),  the  static  signature  does  vary  with  time 
but  with  a  maximum  bandwidth  not  exceeding  a  few 
hertz.  Detection  of  these  fields  from  the  vessel  are  in 
the  near  field  so  the  equations  which  best  describe 
the  propagation  of  the  field  from  the  source  are  the 
near  field  equations  and  not  the  radiation  equations. 
An  interesting  consequence  of  this  is  that  no  energy 
flows  from  the  vessel  in  producing  a  magnetic  field 
at  a  distance  unlike  the  alternating  electromagnetic 
fields  (see  below)  and  the  acoustic  emissions.  (For 
example  at  1Hz,  the  wavelength  in  air  is  300km. 
There  is  insufficient  power  in  frequencies  in  this 
region  for  vessel  detection  at  anywhere  near  this 
distance.) 
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Rotating  equipment  and  alternating  power 
frequencies  also  produce  magnetic  fields,  but  since 
these  are  time-varying  with  respect  to  the  vessel 
they  are  known  as  dynamic  or  alternating  fields. 

Corrosion  of  the  vessel  due  to  different  metals  used 
in  its  construction  (for  example  the  steel  hull  and 
phosphor  bronze  propulsors)  results  in  a  static 
electric  field  in  the  seawater.  This  field,  like  the 
static  magnetic,  only  extends  up  to  a  few  hertz  but  it 
can  be  modulated  by  the  rotation  of  a  propulsor  (for 
example,  by  the  electrical  resistance  of  the  electrical 
path  between  the  shaft  and  the  vessel  through  the 
shaft  bearings)  and  this  produces  higher 
frequencies.  In  addition,  active  cathodic  protection 
devices  (which  operate  by  forcing  a  current  through 
anodes  which  opposes  the  corrosion  current)  often 
have  power  harmonics  present  resulting  in  higher 
frequencies  being  emitted.  Collectively,  the  electric 
field  from  time-varying  sources  on  vessels  is  known 
as  the  alternating  electric  field. 

The  generic  name  for  all  the  underwater 
electromagnetic  fields  is  ELFE  (extra  low  frequency 
electromagnetic). 

Finally,  an  electric  field  cannot  exist  in  seawater  (a 
conducting  medium)  without  an  electric  current  also 
being  present.  An  electric  current  is  always 
associated  with  a  magnetic  field,  so  an  additional 
component  of  the  magnetic  field  is  produced  by  the 
corrosion  current.  This  is  known  as  the  corrosion 
related  magnetic  (CRM)  field. 

Degaussing  Ranges 

Traditionally,  degaussing  ranges  consisted  of  ten  or 
more  underwater  magnetic  sensors  at  a  typical  mine 
threat  depth  (9  to  30m).  Data  collection  was  by 
galvanometer  deflection  and  photographic  film  or 
chart  recorder.  Degaussing  coil  setting  was  by 
successive  approximation,  needing  a  separate  pass 
or  the  vessel  over  the  range  (a  run).  Some  vessels 
have  as  many  as  eighty-two  degaussing  coils  so  the 
number  of  runs  necessary  to  achieve  a  satisfactory 
magnetic  signature  could  take  many  hours.  In  many 
areas,  there  is  significant  tidal  change  over  a  period 
of  hours  so  the  measured  signature  can  vary  by  a 
factor  of  two  or  more.  Furthermore,  the  vessel  was 
unlikely  to  pass  exactly  the  same  distance  from  the 
nearest  sensor  on  successive  runs  further 
complicating  the  analysis. 

In  recent  years,  computer  development  has  allowed 
algorithms  of  some  complexity  to  be  run  on 
processors  which  are  within  the  budget  of 
degaussing  range  operators.  As  a  result,  the 
number  of  underwater  sensors  has  tended  to 
decrease  and  magnetic  modelling  of  the 
magnetisation  of  the  ranging  vessel  has  been  used 
to  simulate  the  additional  sensors  necessary  for 


interpretation  of  the  data.  Modelling  also  removes 
the  variations  caused  by  measurement  at  different 
depths  and  different  distances  from  the  nearest 
sensor.  In  the  UK,  infra-red  tracking,  and  more 
recently,  real  time  kinematic  differential  GPS 
(RTKDGPS)  has  provided  vessel  tracking  to  an 
accuracy  of  approximately  7cm. 

The  UK  Magnetic  Ranges 

All  the  UK  degaussing  ranges  were  replaced  by 
three-axis,  five-sensor,  computer  modelling  ranges 
approximately  ten  years  ago.  Five  years  ago,  the 
ranges  were  upgraded  to  the  present  software 
configuration.  Generally,  the  ranges  have 

performed  well,  with  repeatability  of  typically  1%  or 
so.  But,  on  some  occasions,  anomalous  results  were 
obtained.  The  CRM  field  was  thought  to  be 
responsible  but  separation  of  the  CRM  from  the 
ferromagnetic  field  is  error  prone  because  the  CRM 
is  usually  much  smaller  than  the  ferromagnetic  field 
at  degaussing  range  measurement  distances.  The 
proposed  solution  was  to  measure  the  electric  field 
and  calculate  the  corrosion  currents  and  hence  the 
CRM  field  from  it. 

The  UK  degaussing  ranges  are  located  at  Portland, 
Plymouth,  on  the  Clyde  and  on  the  Forth.  There  is 
also  a  transportable  range  which  can  be  deployed  at 
short  notice  anywhere  in  the  world.  The 
transportable  range  uses  identical  equipment  to  the 
fixed  ranges  which  ensures  commonality  of 
measurement  results. 

One  of  the  benefits  of  the  transportable  range  is  the 
ease  of  deployment  of  the  sensors.  They  weigh  only 
12kg  (in  air)  and  can  easily  be  deployed  by  divers 
using  an  inflatable  boat.  Electric  field  sensors 
usually  consist  of  between  four  and  six  electrodes 
separated  by  between  30cm  and  2m.  If  electric 
sensors  were  to  be  added  to  the  range,  a 
considerable  increase  in  the  weight  of  each  sensor 
would  result  with  a  consequent  increase  in  the 
difficulty  of  deployment. 

If  the  electric  potential  could  be  measured,  only  a 
single  electrode  need  be  used  at  each  sensor 
position  and  no  decrease  in  the  ease  of  deployment 
of  the  transportable  range  would  result. 
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Figure  1  A  magnetic  sensor  with  hydrophone  (top,  or: 

rear)  and  single  electrode  (circled,  upper  right) 


B  =  E.M  (3a) 

Figure  1  shows  a  range  sensor  on  a  transportable 

range  tripod.  On  the  top  of  the  sensor  are  a  while  this  makes  it  possible  to  represent  a  source  of 

hydrophone  (the  tall  extrusion  at  the  back,  an  gny  complexity,  it  is  necessary  to  find  a  magnetic 

electric  potential  sensor  (circled)  and  a  pressure  source  distribution  corresponding  to  a  measurement 

sensor  which  is  present  on  nearly  all  our  magnetic  from  a  degaussing  range.  For  this  a  least  squares  fit 

sensors.  The  sensor  enclosure  is  approximately  jg  used  to  solve  the  inverse  of  equation  3a; 

40cm  high  and  25cm  diameter. 

Static  field  modelling  M  =  (e  .e)  .E  .B  .  (4) 


The  use  of  modelling  techniques  in  magnetic 
ranging  has  been  described  elsewhere  (Ref  1)  so  it 
will  only  be  summarised  here. 

Any  magnetic  source,  no  matter  how  complicated, 
can  be  represented  as  a  collection  of  magnetic 
dipoles.  The  magnetic  field,  B  (actually,  B  should  be 
referred  to  as  the  magnetic  flux  density  but  it  has 
become  conventional  in  the  degaussing  community 
to  refer  to  the  magnetic  field),  from  a  single  dipole  of 
magnetic  moment,  M  is  given  by: 

B  =  — D.M  (1) 

4;r 


where: 

3xy  3xz 

D  =  -^  3xy  3y^ -r^  3yi  (1a) 

^  3xz  3y2  3z^ -r^  ^ 

and  X,  y  and  z  represent  the  vector  components  of 
the  distance  from  the  dipole  source  to  the  field 

measurement  point,  r  =  -Jx^  +  . 

Equation  la  may  be  extended  for  any  number  of 
field  points,  n: 


In  practice,  this  solution  is  unsatisfactory  because  it 
produces  values  of  M  which,  while  they  satisfy  the 
equation,  do  not  provide  a  good  physical 
representation  of  the  distribution  of  the  vessel.  A 
mathematical  technique  for  forcing  the 
magnetisation  to  be  physically  more  realistic  is  given 
in  reference  (1)  and  will  not  be  discussed  further 
here. 

Equations  1  to  4  form  the  basis  of  the  magnetic 
modelling  which  is  used  in  the  UK  magnetic  ranges 
having  been  installed  approximately  five  years  ago. 
As  noted  above,  anomalous  results  have  been 
observed  which  have  been  attributed  to  CRM  fields. 
To  determine  a  source  moment  from  which  the  CRM 
field  can  be  calculated,  an  electric  current  dipole 
distribution  has  to  be  found  by  analogous  methods 
to  the  magnetic  solution  above. 

If  we  neglect  the  seabed  and  the  sea  surface,  the 
equation  for  the  current  density  field,  J,  from  an 
electric  current  dipole  has  exactly  the  same  form  as 
the  equation  for  the  magnetic  field: 

J  =  — D.P  (5) 

4zr 

where  Dis  as  defined  in  equation  la  and  Pis  the 
dipole  current  moment.  The  difficulty  in  adding 
vector  electric  field  measurement  to  a  magnetic 
range  has  been  discussed  above.  Measuring  the 
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electric  potential  provides  an  alternative  that  leads  to 
lighter,  cheaper  sensors  which  have  only  a  single 
additional  electrode. 

If  we  replace  the  current  density,  J ,  in  equation  5  by 
oE  where  <t  is  the  seawater  conductivity  and  E  is 
the  electric  field  and  note  that  the  electric  field  can 
be  written  as  the  gradient  of  an  electric  potential,  we 
can  write: 


— ^D.P  =  E  =  -VF.  (6) 
4^cr 


Integrating  equation  6  gives: 

—  fD.P  =  -F.  (7) 

Atict  •' 


We  can  define  a  vector  Q  = -  D  so  equation  7 

4;7-cr  J 

becomes; 


Q.P=-K.  (7a) 

The  potential,  V  ,  is  not  measurable  but  the  potential 
difference  is  measurable,  so  for  five  sensors  we  can 
write: 

(Q,-Q,).P  =  (F,-F,) 

(Q,-QjP  =  (P,-F,) 

(Q,-Q.).P  =  (^.-^,) 

(Q.-Q,>P  =  (P,-Pj 


where  Qi  represents  the  vector  for  sensor  1  and 

represents  the  potential  difference  between 

sensors  1  and  2.  This  can  be  extended  to  include 
many  measurement  points)  and  many  dipoles  (as  in 
equations  2  and  3  above)  to  represent  an  electric 
polarisation  distribution)  and  then  solved  to  find  the 
polarisation  by  least  squares  fit  as  shown  above 
(equation  4). 

Following  the  analogy  between  electric  field  and 
magnetic  field,  the  electric  field  is  found  from  the 
polarisation  distribution: 

E  =  — D.P  (6) 

Ana 


B 


/^o 


cm 


ATor 


0  ^  -y 

-z  0  X 
y  —X  0 


P 


(9) 


(Note  that  the  equations  given  for  electric  current 
density,  electric  field  and  the  CRM  field  are  all 
simplified  here  assuming  a  sea  with  no  boundaries. 
In  practice,  of  course,  the  sea  is  bounded  by  its 
surface  and  the  seabed.  Above  the  sea  surface  is 
air  which  has  zero  conductivity.  The  seabed 
generally  has  a  finite  conductivity  but  one  which  is 
much  smaller  than  that  of  the  sea.  The  layers  are 
accommodated  by  introducing  a  number  of  images 
of  the  source  dipoles  into  the  basic  equations.) 

Comparison  of  equations  1a  and  9  reveals  that  the 
ferromagnetic  field,  B ,  from  a  magnetic  dipole 

decreases  as  —  whereas  the  CRM  field,  ^cm 
r 

decreases  as  .  Although  the  CRM  field  is  much 

smaller  than  the  ferromagnetic  field  at  typical 
degaussing  range  measurement  distances,  it 
becomes  relatively  bigger  as  distance  increases. 


Figure  2  Comparison  between  ferromagnetic  field 
(solid  line)  and  CRM  field  (dashed  line)  with  distance 

Figure  2  shows  the  comparison  between  the 
ferromagnetic  field  and  the  CRM  field  with  distance. 
In  this  example  (using  untypical  distances  and  fields 
so  it  is  unclassified)  the  CRM  field  becomes  equal  to 
the  ferromagnetic  field  100m  from  the  source  yet,  at 
25m  from  the  source,  the  ferromagnetic  field  was 
five  times  the  magnitude  of  the  CRM  field. 

Although  corrosion  currents  cannot  penetrate  the  air 
(since  it  has  zero  electrical  conductivity)  they  extend 
for  large  distances  in  the  sea  so  the  magnetic  field 
from  them  extends  into  the  air. 


We  need  to  find  the  CRM  field  and  that  is  given  by: 


Airborne  Detection  (MAD) 

Some  aircraft  are  equipped  with  sensitive 
magnetometers  (usually  in  a  tail  boom  or  deployed 
on  a  wire)  with  which  submerged  submarines  may 
be  detected.  As  the  technology  was  first  developed 
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for  geological  surveying  for  Magnetic  Anomaly 
Detection  (MAD),  the  technique  of  detecting  vessels 
from  the  air  is  commonly  referred  to  as  MAD.  The 
magnetic  field  from  a  distant  submarine  is  much 
smaller  than  the  Earth’s  magnetic  field  and,  indeed, 
smaller  than  the  fields  resulting  from  the  interaction 
of  the  Earth’s  magnetic  field  with  the  aircraft. 
Compensation  of  the  airborne  magnetometer  for 
motion  of  the  aircraft  is  thus  necessary. 

Hitherto,  a  vessel’s  magnetic  field  has  been  referred 
to  as  a  vector  field.  Measurement  of  the  components 
of  a  vector  field  requires  can  only  be  made  in  the  co¬ 
ordinate  frame  of  the  measurement  platform.  If  the 
measurement  platform  has  any  rotational  motion, 
the  Earth’s  magnetic  field  will  be  measured  as 
rotating  in  the  opposite  direction.  As  flying  aircraft 
always  have  some  rotational  moment  (for  example 
rolling  and  yawing)  an  apparent  rotating  field  is 
always  measured.  The  apparent  rotating  field  Is 
much  larger  the  field  measured  from  a  submarine  at 
a  distance,  so  compensation  for  it  is  not  currently 
possible.  Therefore,  it  is  customary  to  measure  a 
property  of  the  field  which  is  rotationally  invariant. 

For  a  vector  field,  the  magnitude  +8^+3^  ) 

is  invariant  with  rotation  so  MAD  sensors  effectively 
measure  the  scalar  field.  To  determine  the  MAD 
threat  to  a  vessel,  the  degaussing  range  software 
has  to  calculate  the  field  which  may  be  detected  by 
an  airborne  sensor.  It  may  be  thought  that  it  is 
simply  necessary  to  calculate  the  magnitude  of  the 
predicted  magnetic  field  at  an  appropriate  distance. 
The  following  discussion  will  show  why  this  is  not  the 
case. 


Be 


4bI  +  bI 


- ►-> 

Be  By 

Figure  3  Vessel’s  field  normal  to  Earth's  field  (upper) 
and  parallel  to  Earth's  field  (lower  figure) 

Figure  3  illustrates  the  problem.  The  upper  diagram 
shows  the  case  where  the  vessel’s  field  happens  to 
be  at  right  angles  to  the  Earth’s  field.  The  resultant 

magnitude  is  given  by  -f  By  (the  dashed  line 

in  the  figure).  If  By  is  very  much  less  than  B^  then 

■^JbI  -h  By  «  B^ .  For  example,  it  the  vessel’s  field 


is  10nT  and  the  Earth’s  field  is  50,000nT 
(nanotesla),  yjB^By  is 

VSOOOO^+IO^  =  50000.001  nT.  The  sensor 
would  need  a  resolution  of  0.001  nT  to  detect  a  lOnT 
field.  In  the  lower  diagram,  where  the  two  vectors 
are  parallel,  their  sum  is  simply  the  sum  of  their 
magnitudes:  50000-1-10  =  50010  nT.  (Note  that  the 
values  for  a  vessel’s  MAD  field  given  here  are  not 
representative  of  any  particular  vessel  but  are 
chosen  to  illustrate  the  problem  and  to  be 
unclassified.)  Effectively,  the  scalar  sensor  on  a 
moving  platform  in  the  Earth’s  magnetic  field  detects 
only  that  component  of  the  vessel’s  signature 
parallel  to  the  Earth’s  field. 

In  air,  the  vessel’s  magnetic  field  consists  of  the  sum 
of  the  ferromagnetic  field  and  the  CRM  field  but  the 
only  part  of  this  detectable,  ,  is  the  component 
parallel  to  the  Earth’s  magnetic  field.  This  is  given  by 
the  scalar  or  dot  product  of  the  two  vectors: 

^DET  “  ®£-(®F£/lRO 

The  measured  magnetic  field  consists  of  the 
ferromagnetic  field  plus  the  CRM  field,  so  accurate 
modelling  of  the  ferromagnetic  field  requires  removal 
of  the  CRM  field  from  the  measurement.  While  the 
CRM  field  is  generally  smaller  than  the 
ferromagnetic  field  at  typical  degaussing  range 
distances,  it  is,  for  many  vessels,  by  no  means 
negligible.  Therefore,  the  CRM  field  at  the 
measurement  sensor  positions  is  calculated  from  the 
electric  potential  measurements,  subtracted  from  the 
magnetic  measurements,  and  the  ferromagnetic 
model  calculated  from  the  result.  This  ensures  that 
electric  and  magnetic  source  models  of  sufficient 
accuracy  exist  for  the  predictions  described  abovr  to 
be  valid. 

UK  Multi-influence  range  description 

The  UK  multi-influence  ranges,  whether  in  the  fixed 
or  transportable  configuration,  consist  of  five 
underwater  sensors  from  Ultra  (designated 
MUWS10  and  MUWS10A,  depending  on  whether  an 
acoustic  sensor  is  fitted).  The  original  magnetic 
ranges  used  Ultra’s  MUWS3  and  MUWS3P  which 
had  serial  digital  communications  with  the  shore.  At 
the  present,  about  seventy  of  these  are  still  In  use 
around  the  UK.  The  MUWS10  design  reverted  to 
analogue  communication  because  the  potential 
differences  between  pairs  of  sensors  has  to  be 
measured  by  differential  amplifiers  in  an  underwater 
junction  box.  Furthermore,  the  sensitivity  of  the 
potential  difference  inputs  is  of  the  order  of  5nV  so 
digital  signals  within  the  sensor  were  not  desirable  to 
avoid  electromagnetic  compatibility  (EMC)  problems. 


16.5 


The  underwater  junction  box  (UJB)  is  quite 
complicated.  It  receives  analogue  signals  from  five 
underwater  sensors  and  divides  the  magnetic  and 
potential  sensor  outputs  into  two  frequency  bands  - 
0  to  5Hhz  and  5Hz  to  3kHz.  Each  band  has 
analogue  low-pass  filtering  to  ensure  out  of  band 
signals  do  not  cause  aliasing.  As  each  sensor 
produces  three  static  magnetic  outputs,  one 
pressure  and  one  electric  potential  plus  three 
alternating  magnetic  and  one  alternating  potential, 
nine  inputs  per  sensor  are  required  at  the  UJB  -  a 
total  of  45  inputs.  Three  complete  channels  are 
provided  for  future  enhancement. 

Foliowing  the  analogue  data  acquisition  and  signal 
conditioning  channels,  the  static  data  digitised  by  24 
bit  A  to  D  converters  and  the  alternating  data  by  16 
bit  a  to  D  converters.  All  the  digital  data  is  buffered 
before  being  packaged  onto  a  serial  data  stream 
and  transmitted  to  the  shore  using  an  ADSL  modem. 
This  has  been  designed  to  enable  the  use  of  the 
existing  underwater  cables  -  future  enhancements 
include  conversion  to  fibre-optic  cables. 

On  the  shore,  data  is  archived  to  a  networked 
72Gbyte  hard  disk  drive  from  which  separate  data 
processing  computers  for  static  fields,  alternating 
fields  and  acoustic  data  receive  their  information. 

An  underwater  junction  box  is  shown  on  its  glass 
reinforced  plastic  mounting  in  figure  5  just  prior  to 
deployment.  It  is  approximately  0.5m  in  diameter 
and  0.5m  long  and  weighs  approximately  70kg  in  air. 


Figure  5  Underwater  Junction  Box 


The  UJB  housing  is  titanium  to  minimise  corrosion 
(which  would  also  compromise  the  integrity  of  the 
underwater  electric  potential  measurements)  and  to 
maximise  heat  transfer  from  the  internal  electronics 
to  the  sea. 

Data  Analysis 

The  static  field  data  analysis  is  by  display  of  the 
vector  field  compensated  for  water-depth  and  sensor 


position  variations.  The  CRM,  ferromagnetic  and 
electric  fields  can  each  be  displayed  for  the  operator 
to  assess  the  threat.  Typical  displays  are  shown  in 
figures  6,  7  an  8  for  illustrative  purposes  only  since  it 
is  difficult  to  interpret  these  in  monochrome. 

In  these  figures,  the  left  hand  panel  shows  the 
signature  along  the  vessel  over  two  vessel-lengths, 
with  the  bow  to  the  left  and  the  right  hand  panel  the 
athwartships  cross  section  over  three  vessel-widths 
with  port  to  the  left.  The  three  curves  on  each 
display  indicate  longitudinal,  athwartships  and 
vertical  field  components  which  are  signified  on  the 
computer  displays  by  different  colours  which  cannot 
be  seen  here. 

Ferro  EWEOQ107  Heading  310  SCALED 

Woter  depth  9.00  Date  26  FEB  1999 

Draught  2.20  Time  09:07:10  Distance  from  keel  -4.50 

Moximum  Total  1445  Distonce  from  midships  0.00 

L:  -79.6  A:  108  V:  1438  T:  1445 


Distance  from  midships  Distonce  from  keel 

Figure  6  A  ferromagnetic  signature 

CRM  EWE00107  Heading  310  SCALED 

Water  depth  9.00  Date  26  FEB  1999 

Drought  2.20  Time  09:07:10  Distonce  from  keel  0.00 

Maximum  Total  304  Distance  from  midships  -21.2B 

L;  11.1  A:  -289  V:  -72.3  T:  29B 


Distance  from  midships  Distance  from  keel 

Figure?  The  CRM  signature 

Figures  6,  7  and  8  are  for  actual  vessels  but  they 
have  been  scaled  to  make  them  unclassified.  The 
scale  factor  is  constant,  so  it  is  clear  from 
comparison  of  figures  6  and  7  that  the  CRM 
signature  is  approximately  25%  of  the  ferromagnetic 
signature  -  by  no  means  negligible.  It  is  also  evident 
that  the  CRM  signature  is  concentrated  near  the 
stern  of  the  vessel  as  would  be  expected  since  the 
corrosion  currents  are  concentrated  around  the 
propulsor. 


Elec  EWE00107 
Water  depth  9.00 
Drought  2.20 
Maximum  Total  28355 
L:  -3032 


Heoding  310  SCALED 

Dote  26  FEB  1999 

Time  09:07:10  Distance  from  keel  0.00 
Distance  from  midships  0.00 
A:  610  V:  -812  T:  3197 


Distance  from  keel 


Distance  from  midships 


Figure  8  The  electric  signature 

As  the  electric  signature  and  the  CRM  signature  are 
interdependent,  it  is  not  surprising  that  both  show  a 
maximum  around  the  stern  of  the  vessel. 

Figures  9,  10  and  11  show  the  calculated  in-air 
signatures  corresponding  to  the  underwater 
signatures  shown  in  figures  6,  7  and  8. 


Figure  9  MAD  signature  due  to  ferromagnetic  and 
CRM  fields 

Figure  9  shows  the  MAD  signature  due  to 
ferromagnetic  and  the  CRM  field.  This  is  the  field 
which  would  be  detected  by  a  MAD  sensor  with  a 
sensitivity  (after  compensation  for  the  aircraft 
motion)  of  InT.  This  is  shown  150m  above  the 
vessel  and  for  a  1km  square  centred  on  the  vessel. 
Where  the  signature  is  detectable,  the  contour  lines 
are  shown  in  dark  grey  -  light  grey  lines  show  an 
undetectable  signature.  (As  with  the  other  displays, 
the  height,  signature  and  threshold  of  detection  are 
not  representative  of  actual  vessels  but  are  shown 
for  illustrative  purposes  only  so  the  data  is 
unclassified.) 


Figure  10  The  MAD  signature  due  to  the 
ferromagnetic  field  only 

Figure  10  shows  the  MAD  signature  due  solely  to 
ferromagnetic  sources.  This  is  unmeasurable  and 
can  only  be  produced  by  calculation.  The  difference 
between  figures  9  and  10  is  obvious. 

Finally,  figure  11  show  the  MAD  signature  due  to 
CRM.  This  is  also  unmeasurable  and  can  only  be 
produced  by  computation. 


Figure  11  The  MAD  CRM  signature 

The  CRM  signature  alone  from  this  vessel  is  not 
detectable  with  a  InT  sensitivity  but  the  effect  of  the 
CRM  signature  in  modifying  the  gross  signature  is 
obvious  from  the  differences  between  figures  9  and 
10. 

Validation  of  results 

Validation  of  the  accuracy  of  the  determination  of  the 
CRM  field  for  actual  vessels  is  virtually  impossible 
due  to  the  large  ferromagnetic  field  which  is 
practically  always  present.  Fortunately,  a  trial  took 
place  a  few  years  ago  where  a  mine 
countermeasures  vessel,  with  a  very  low 
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ferromagnetic  signature,  was  ranged  at  30m  depth 
passing  over  both  an  electric  field  sensor  and  a 
range  with  five  electric  potential  sensors.  As 
described  above,  the  CRM  field  was  calculated  from 
the  electric  model  derived  from  the  potential 
measurement  and  then  compared  with  the 
measured  magnetic  measurement.  The  result  is 
shown  in  figure  12. 


Figure  2  CRM  field  predicted  from  electric  potential 
measurements  compared  with  measured  CRM  field 

The  close  agreement  between  each  of  the  three 
axes  is  clearly  seen.  Close  examination  of  the  figure 
shows  that  one  of  each  axis  is  noisier  than  the  other 
-  that  is  the  measured  CRM  as  it  has  the  Earth’s 
background  noise  present  whereas  the  predicted 
CRM  does  not. 

ELFE 

The  ELFE  analysis  and  displays  are  not  formalised 
at  the  present  time.  When  these  are  implemented, 
the  results  will  form  the  contents  of  another  paper  if 
their  security  classification  allows. 

Summary 

The  upgrade  of  the  UK  degaussing  ranges  from 
magnetic  measurement  only  to  multi-influence 
electromagnetic  measurement  has  been  described. 
Signatures  have  been  shown  for  illustrative 
purposes  only  but  it  is  clear  from  these  that  the 
corrosion  related  magnetic  field  cannot  be  ignored 
for  threat  evaluation  -  whether  underwater  or  from 
airborne  detection  (MAD). 
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Abstract 

The  Land-Based  Test  Site  facilities  at 
DERA  Portsdown,  are  used  for  the 
integration,  proving  and  acceptance  of  the 
Combat  Suite  for  the  Navy’s  Type  42 
Destroyer,  Aircraft  Carrier,  MCMVs  and 
Type  23  Frigates.  The  facility  combines 
real  combat  system  elements,  sensors,  C3 
and  weapons,  with  a  range  of  simulators 
and  stimulators,  which  include  the  ability  to 
create  complex  real  time  scenarios. 
Integration  of  hardware  and  software 
combat  system  elements,  which  have  to 
work  robustly  in  a  real  time  environment,  is 
notoriously  difficult  and  the  facilities  have 
been  shown  to  have  saved  its’  costs  many 
times  over  in  the  last  20  years. 

Technology  is  moving  on  and  the  next 
generation  of  Shore  Integration  Facilities  is 
likely  to  be  more  distributed,  with  combat 
system  elements  on  separated  sites, 
connected  by  high  capacity  networks. 
Future  systems  will  be  even  more  complex 
and  the  integration  and  proving  issues  are 
extending  from  Platform,  to  Force,  to  Joint 
Service  interoperability.  DERA  is 
developing  the  Land  Based  Test  Site  into 
a  unit  that  can  help  to  solve  these 
problems.  The  goal  is  for  it  to  become  the 
National  Maritime  Reference  Centre,  with 
high  speed  links  to  other  such  test  centres 
e.g.  the  Land  Reference  Centre  at 
Blandford  and  other  DERA,  military  and 
contractor  sites,  and  to  have  the 
necessary  networked  simulator/stimulator 
facilities  to  create  the  complex  “virtual 
world”  needed  to  exercise  and  prove  such 
systems. 

1. Introduction 

1,1  Overview. 

Portsdown  Hill  is  a  chalk  escarpment  120 
metres  high.  It  has  an  excellent  vantage 
point  with  spectacular  views  across  the 
sea  to  the  Isle  of  Wight.  However  it  is 
much  more  than  just  a  view. 

Portsdown  Hill  has  an  interesting 
military  history,  as  well  as  being  an 


important  area  for  wildlife,  there  is  much 
more  to  explore.  Combat  System 
Integration  takes  place  at  DERA  Land- 
Based  Test  Site.  This  paper  describes  the 
facilities  at  the  Land-Based  Test  Site 
(LBTS)  Portsdown,  which  are  used  for 
integration,  proving  and  acceptance  of  the 
Combat  Suite  for  the  Royal  Navy’s  Type 
42  Destroyer,  Aircraft  Carrier,  and  the 
Type  23  Frigate.  The  paper  gives 
examples  of  typical  integration,  testing  and 
acceptance  problems  and  how  they  have 
been  addressed. 

The  paper  discusses  the  latest  plans  for 
the  next  generation  of  Shore  Integration 
Facilities  (SIF),  for  future  platforms  such 
as  the  Type  45  Destroyer.  We  discuss  the 
vision  for  the  Land-Based  Test  Site,  for  it 
to  become  the  National  Maritime 
Reference  Centre,  and  how  this  will  be 
achieved. 

1.2  Background 

The  difficulty  of  integrating  Combat 
System  equipments  on  modern  warships 
has  been  recognised  for  some 
considerable  time.  Jordan  Lee  and 
Cawsey  in  their  ‘Learning  From 
Experience  report on  the  issues  of 
integrating  Software  Intensive  Projects 
within  Naval  platforms  and  the  good 
practise  in  the  use  of  Shore  facilities 
quoted;-  “There  is  already  a  facility  at 
Portsdown  -  in  effect  a  concrete  ship  -  in 
which  system  integration  can  be 
demonstrated  and  tested.  These  facilities 
provide  improved  opportunities  for  difficult 
issues  of  system  integration  in  platform 
projects  to  be  addressed  early.” 


'  Arrangements  for  Managing  Major  projects  in 
MoD(PE)/1987 
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The  Staff  Requirement  (SR)  for  the  Type 
23  Frigate  includes  the  requirement  for  the 
Type  23  SIF.  The  Type  23  Shore 
Integration  Faciiity  was  set-up  at  the  Land- 
Based  Test  Site  in  accordance  with  the 
requirement,  with  the  intention  of  providing 
a  realistic,  flexible  test  bed  to  enable  the 
efficient  deveiopment,  iinking,  integration 
and  proving  of  the  Type  23  Combat 
System.  The  Type  23  SIF  is  the  most 
comprehensive  facility  at  the  LETS. 

The  primary  functions  of  the  Type  23  SIF 
are  to  trial  and  prove  the  interfaces 
between  co-operating  weapon  systems, 
the  proving  of  weapon  and  sensor 
operational  software  and  the  Human- 
Computer  Interfaces  (HCI).  The  SIF 
provides  reproducible  scenario 
environments  tailored  to  engineering  tests 
together  with  operational  scenarios 
necessary  for  the  integration  and 
evaluation  of  the  Combat  System.  Load 


possible.  We  don’t  have  the  guns, 
missiles  and  sonar  transducers,  but 
simulate  these. 

•  We  use  operational  software  in  all  our 
systems. 

•  We  stimulate  subsystem  equipments 
as  near  to  the  data  or  signal  input 
source  as  practicable. 

•  We  drive  stimulators  from  a 
centralised  Scenario  Generator. 


Combat  System  Equipment  (CSE) 

There  is  extensive  range  of  CSE  installed 
at  the  LETS.  The  Type  T23  SIF  has  the 
majority  of  the  Combat  System  elements 
that  you  would  find  fitted  on  an  operational 
ship.  There  are  exceptions  to  this,  there 
would  be  no  benefit  fitting  sonar  arrays, 
and  so  only  inboard  equipment  is  installed 


testing  can  also  be  carried  out  under 
conditions  of  stress,  which  cannot  be 
reproduced  in  peacetime  exercises.  In  the 
longer  term,  the  Type  23  SIF  will  provide  a 
main  site  for  post  development  support 
and  testing  and  evaluation  of  the 
responsive  adaptation  programmes, 
thereby  reducing  the  demand  for  time  on 
operational  ships. 

1.3  Facilities  -  Design  Philosophy 

This  section  describes  the  range  of 
equipments  available  to  the  users  of  the 
SIF,  how  they  are  connected  together,  and 
simulated  and  stimulated  in  a  real  time 
environment.  The  design  philosophy  for  all 
our  Shore  Integration  Facilities  can  be 
broken  down  as  follows :- 
•  We  fit  as  much  of  the  real  Combat 
System  Equipments  subsystems  as 


Figure  1  Typical  CSH  Configuration 


at  the  LETS.  The  Command  System  for 
the  Type  23  Frigate,  (DMA)  is  linked  to  the 
other  elements  of  the  Combat  System  via 
a  Combat  System  Highway  (CSH).  Two 
complete  Combat  System  Highways  are 
installed  at  the  SIF.  These  are  poll  and 
response  highways  based  on  the  DEF 
STAN  0019,  giving  a  high  level  of  service 
and  redundancy.  Triplicated  highways  are 
implemented  with  access  via  active 
remote  highway  couplers  and  spur  cables, 
to  each  CSE.  Each  highway  has  two 
Highway  Controllers  Health  Monitors 
(HCHM)  which  control  member  system 
polling  and  health  monitoring  facilities. 
There  are  still  a  few  legacy  equipments 
fitted  that  use  point  to  point  interfaces,  and 
therefore  are  not  connected  to  the  CSH. 
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Figure  1  shows  the  typical  configuration  of 
a  Type  23  Combat  System  Highway. 

The  Type  42  SIF  was  installed  when  there 
was  a  different  philosophy  to  integration, 
and  also  severe  cost  constraints.  Most  of 
the  real  CSE  installed  on  Type  42 
Destroyers  have  been  emulated  at  the 
SIF.  Again  exceptions  can  be  found, 
radars  are  fitted,  and  the  T996  Radar  can 
be  shared  with  the  T23  SIF.  The  Type  42 
long  range  surveillance  radar,  and  the  Sea 
Dart  tracker,  are  fitted  at  the  LBTS,  and 
are  now  the  reference  equipment  for  those 
projects. 

The  LBTS  also  has  a  number  of  data  link 
networks.  This  enables  users  to  configure 
systems  so  that  different  platforms  can  be 
integrated.  Land  lines  enable  a  wide  area 
network  to  be  configured,  so  a  ship  in  port 
can  be  included  in  a  scenario  run  from  the 
LBTS.  The  configurable  system  gives 
users  the  facility  to  run  trials,  with  different 
platforms,  on  the  same  site. 

Simulation  Equipment 

There  is  continuing  debate  regarding  the 
use  of  simulation  for  integration  purposes 
as  against  stimulation  of  real  equipment 
when  implementing  a  SIF.  As  mentioned, 
before  the  T23  SIF  is  based  on  stimulating 
real  equipment,  as  opposed  to  the  T42 
where  simulation  is  used.  Both,  however, 
have  the  capability  of  running  a  scenario  in 
real  time.  The  Scenario  Generators  (SGs) 
are  capable  of  modelling  hundreds  of 
platforms,  all  manoeuvring.  The  playing 
areas  are  in  excess  of  8000Km  square, 
and  32000m  height  and  depth.  There  are 
many  parameters  that  can  be  modelled, 
such  as  environmental  conditions,  e.g. 
wind,  tide,  sea  state,  clouds,  and  acoustic 
conditions  e.g.  sea  depth,  bottom  type, 
shadow  and  zones.  Emitters  such  as, 
radars.  Jammers,  and  sonars,  can  be 
attached  to  the  platforms.  The  SGs  are 
driven  from  a  script,  which  gives  the  ability 
to  loop  the  scenario  giving  an  infinite  run 
time.  To  enable  feedback  into  the 
scenario,  information  on  weapon 
deployment  is  sent  to  the  SGs.  The  SG 
can  then  remove  a  hit  platform  from  the 
scenario  if  required. 

Within  the  T23  SIF  there  are  two  Scenario 
Generator  Highways.  These  are  based  on 
the  same  principle  as  the  CSH’s  except 
they  are  dual  redundant  only.  Stimulators 


are  connected  to  these  in  the  same 
manner  as  real  equipment. 

Stimulators  are  designed  in  a  manner  to 
interface  with  the  real  equipment,  as  close 
to  the  first  digital  point  in  the  system 
processing  chain  as  practical.  This 
maximises  the  stimulation  of  the  real 
equipment  hence  giving  a  more  realistic 
integration  facility. 

Recording  and  Analysis  Equipment 

The  LBTS  has  an  extensive  range  of 
recording  and  analysis  equipment.  There 
are  several  key  systems  in  this  area.  Data 
And  Event  Logging  System  (DAELOS),  is 
connected  in  a  manner,  so  recording  of  all 
broadcast  messages  from  both  CSH  & 
SGH  are  recorded,  and  time  stamped. 
This  gives  the  user  the  ability  to  cross 
reference  events  from  SGH  to  CSH. 
DAELOS  is  configured  so  that  unlimited 
recording  is  available,  provided  periodic 
back-up’s  are  carried  out. 

If  required  the  data  can  be  transferred, 
filtered  and  then  converted  in  to  a  format, 
(s  record),  which  a  DERA  designed 
analysis  tool  can  then  use.  Data  Analysis 
System  for  Highways  (DASH),  gives  the 
user  the  ability  to  look  at  one  particular 
message  in  easy  to  comprehend  graphical 
and  tabular  forms. 

Data  manipulation  and  display  facilities  are 
provided  to  suit  particular  requirements  of 
individual  messages.  Without  these  tools 
integration  of  Combat  Systems  would  not 
be  practicable.  These  tools  are  essential 
to  support  of  Combat  System  Integration. 
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Figure  2.  Typical  SGH  Configuration 


Figure  2,  shows  part  of  the  configuration  of 
the  SIF.  The  Scenario  Generator 
broadcasts  track  and  ship  navigation 
information  to  simulators/stimulators,  via 
the  SGH.  This  is  then  passed  to  the  real 
CSE  equipment,  which  is  linked  to  the 
CSH.  Whist  this  is  happening  HCHMs  are 
monitoring/controlling  highway  messages. 
DAELOS  is  connected  to  both,  to  record 
all  messages. 


1.4  Intearation  Testing  and  Acceptance 
Philosophy 

This  process  is  briefly  explained  by  the 
following. 

Once  the  equipment  has  passed  the 
Factory  Acceptance  Trial  (FAT),  it  would 
then  be  installed  at  the  SIF.  An  installation 
inspection  would  then  be  carried  out 
based  on  documents,  produced  for  ship 
fitting.  The  Royal  Navy’s  Captain  Weapon 
Trials  Authority  (CWTA)  would  then 
inspect  the  installation.  If  any  installation 
problems  were  found  these  would  be 
passed  back  to  the  CSE  project  to  correct 
before  the  equipment  is  fitted  on  a  ship  , 
thus  saving  valuable  ship  time.  After  this 
stand  alone  functional  tests  would  be 
carried  out.  The  next  key  step  would  be  for 
the  equipment  to  under  go  linking  trials. 
This  process  is  mandatory  for  all  systems 
connected  to  the  CSH.  The  CSH  simulator 


gives  the  Combat  System  Engineers  the 
ability  to  check  that  the  equipment  is 
operating  in  accordance  with  the  highway 
protocol  and  will  not  corrupt  other 
messages  on  the  highway.  Once  the 
member  system  has  gained  the  necessary 
certification,  it  is  then  permitted  to  connect 
to  the  CSH  for  full  scale  system  integration 
to  commence. 


2.  Problems  that  Typical  Combat  System 
Integration  have  highlighted 

In  line  with  smart  procurement  these  large 
software  intensive  projects  are  procured 
by  an  incremental  acquisition  approach, 
i.e.  the  software  is  developed  in  a  number 
of  phases,  taking  maybe  4-6  phases  to 
complete  the  build.  On  average  we  expect 
approximately  1 000  pick  up  items  on  each 
phase.  These  items  range  from  software 
faults,  incorrect  specifications  and  Combat 
System  design  problems. 

Below  are  examples  of  anomalies  found. 
Some  of  which  would  have  had 
catastrophic  consequences  if  not 
corrected. 

1)  Potentially  threatening,  fast,  inbound 
targets  are  uniquely  identified  with  a  track 
number.  This  track  would  then  be  passed 
to  the  Command  System,  which  would 
then  carry  out  Threat  Evaluation  Weapon 
Assignment  (TEWA).  In  this  instance  the 
operator  confirmed  the  track  as  “Friendly”. 
The  aircraft  was  then  tracked  outbound 
from  the  ship.  It  then  turned  inbound  for  a 
second  run  where  it  should  have 
maintained  the  “Friendly”  identification  - 
but  didn’t.  It  was  re-allocated  as  “Hostile”. 
The  air  defence  weapon  system  then 
carried  out  a  simulated  engagement  , 
much  to  everyone’s  surprise.  A  solution 
was  identified  and  implemented  within  a 
new  issue  of  software. 

A  further  trial  was  arranged  and  the  SIF 
was  again  used  to  repeat  the  trial  and 
confirmed  that  the  software  modification 
now  ensured  that  friendly  tracks  would  not 
be  engaged  by  the  weapon  system. 

2)  A  recent  build  of  Command  System 
software,  was  found  to  have  a  problem 
displaying  tracks  on  to  the  Multi-function 
console  (MFC).  It  was  found  there  was  a 
difference  of  up  to  a  minute  between 
tracks  being  displayed  on  the  planned 
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position  indicators  (PPl)  displays,  and  the 
MFCs.  The  consequence  of  this  was  that 
operators  actions  and  command  decision 
were  not  effective.  This  could  have  meant 
a  missile  might  have  hit  the  ship  before  the 
operator  had  time  to  respond  with  counter 
measures.  Following  a  software 
modification  and  retrial  at  the  SIF  this  fault 
has  been  rectified. 

3)  Command  Systems  often  under  go 
contractual  reliability  trials  at  the  SIFs. 
These  trials  stress  the  Combat  System 
under  high  load  scenarios  for  long  periods 
of  time.  It  would  be  impossible  to  achieve 
such  a  comprehensive  trial  onboard  ship. 

As  an  aside,  it  is  worth  noting  the 
approximate  amount  of  source  code 
contained  within  Naval  Command 
Systems.  Table  shows  the  number  of 
faults  per  lines  of  code  you  would  expect 
to  find  in  a  typical  software  project.  You 
can  see  from  the  table  the  number  of 
faults  found  for  both  small  and  large 
systems.  .  A  ship  is  a  large  software 
system  which  is  why  progressive  software 
developing  and  testing  at  a  SIF  is  so 
important  if  reliable  systems  are  to  be 
delivered  to  the  Fleet. 


Pre  code 
Generation 

Fault  Density* 

(per  1000  lines  of  code) 

Estimated  faults 

Low  1  High 

Low 

High 

670,000 

670 

5628 

To  1  8.4 

1,770,000 

1770 

14,868 

♦Fault  density  depend  on 

•  Complexity  of  code 

•  Skill  of  company 

•  Size  of  product 

•  Novelty  of  product 


This  section  lists  some  of  the  lessons 

learnt. 

•  Incorporation  of  real  equipment  is 
essential  for  system  integration.  Use 
of  equipment  simulation  is  limited  to 
integration  de-risking  and  operator 
training.  However  if  stimulation  of  real 
equipment  is  used  this  can  be  real 
time  and  repeatable. 

•  The  life  of  a  SIF  will  be  longer  than 

that  of  its  components,  so 
obsolescence  will  always  be  a 

problem.  The  use  of  Commercial  Off 

The  Shelf  (COTS)  hardware  and 
software  does  not  always  provide  a 
solution  and  brings  its  own  unique 
problems. 

•  Standardise  hardware  as  far  as 

possible  and  ensure  software  is 

written  with  long-term  support  in  mind, 
(e.g.  no  clever  system  calls,  which  will 
not  work  with  the  next  version  of 
operating  system.) 

•  The  ability  to  record  all  data  for  the 

duration  of  the  longest  scenario  run 

required  is  essential.  Choice  of 

recording  medium  is  important 
because  of  the  large  amount  of  data 
that  needs  to  be  recorded. 

•  The  essential  analysis  function  is  rapid 
selection  of  large  amounts  (gigabytes) 
of  recorded  data  with  comprehensive 
filtering  criteria. 

•  Users  should  be  able  to  set  up  their 
own  ad-hoc  analysis  procedures  by 
selecting  and  combining  a  set  of  basic 
operations  since  it  has  proved  very 
difficult  to  predict  what  they  will  need 
to  do  when  evaluating  system 
anomalies. 


Table  1  Software  faults. 


3  Learning  From  Experience 

Over  the  years  we  have  learnt  a  lot  from 
Integration  of  different  Command  Systems 
and  the  way  the  SIF  operates.  We  have 
seen  the  simulation  and  stimulation 
philosophy  change  as  the  requirements  for 
integration  and  testing  evolve.  Users  are 
now  demanding  more  functionality, 
reliability,  capability  and  interoperability. 


"  Software  faults  table.  HE  publication. 


4  Future  Platforms 

□ERA  have  research  programmes  in 
support  of  many  future  platforms  which 
include,  the  Future  Aircraft  Carrier  (CVF), 
the  Future  Surface  Combatant  (FSC),  the 
Future  Attack  Submarine  (FASM)  and  the 
Type  45  Destroyer. 

The  Type  45  Anti  Air  Warfare  (AAW) 
Destroyer  is  required  to  give  the  UK’s 
Royal  Navy  a  destroyer  with  specialist 
AAW  capability.  It  will  replace  the 
capability  of  the  Type  42  Destroyer,  a 
platform  which  is  now  suffering  from 
obsolescence  and  is  costly  to  maintain. 
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DERA  have  been  involved  \with  research 
programmes  supporting  the  Type  45 
Destroyer  and  believe  they  could 
contribute  to  it’s  integration  v\rith  the 
Principal  Anti-Air  Missile  System 
(PAAMS),  and  the  Multi-Function  Radar 
(MFR)  SAMPSON.  DERA  are  currently  in 
negotiation  with  the  Type  45  and  PAAMS 
projects  on  the  possibilities  of  installing  a 
test  and  integration  facility  at  DERA 
Fraser,  an  over  water  range  located  on  the 
shore  of  the  Solent.  The  location  is 
representative  of  the  environment  that 
sensors  would  be  expected  to  operate 
under.  A  secure  data  link  to  the  LETS 
would  provide  integrated  sensor  and 
Combat  System  Integration  facilities. 

5  What  next?  New  Generation  Shore 
Integration  Facilities. 

With  new  technology  being  developed  at 
an  alarming  rate  the  way  SIFs  are 
designed  must  change.  The  new 
generation  of  Shore  Integration  Facilities 
will  have  to  be  adaptable  not  only  to  new 
technology  but  be  responsive  to  change  at 
short  notice  although  long  term  support  to 
many  platforms  will  still  be  required.  With 
partnerships  with  industry  and  other 
countries  ever  increasing  we  need  to  look 
at  networking  to  other  integration  and 
reference  sites  in  the  UK  and  overseas. 


Networks  (WANs).  These  are  being  looked 
at  for  a  possible  replacement  CSH  and  for 
connection  of  different  integration  sites. 

DERA  has  also  been  supporting  industry 
in  trials  with  legacy  systems,  integrating 
them  with  Fibre  Optic  technologies. 

The  likely  COTS  basis  of  future  Combat 
Systems,  and  particularly  the  likely  use  of 
‘Middleware’  between  the  applications  and 
the  networking  technology,  has  moved  the 
emphasis  away  from  full  military  networks 
towards  the  use  of  COTS  networks,  or  at 
the  very  least,  the  use  of  standard 
operating  system  network  drivers. 

The  emerging  Gigabit  Ethernet  switched 
LAN  technology  is  part  of  the  CSI  research 
programme  along  with  ATM 
(Asynchronous  Transfer  Mode).  It  is 
anticipated  that  the  Gigabit  Ethernet  will 
support  the  data  rates  and  bandwidths  that 
a  future  platform  will  require. 


5.1  Networks 


Future  naval  combat  systems  will  demand 
more  from  the  Combat  System  Local  Area 
Networks  (LANs).  Where  the  current  LAN 
was  only  designed  to  handle  tactical  real 
time  data  it  is  likely  that  in  the  future  the 
highway  will  need  to  support  other 
services  like  audio  and  video.  This 
requires  increasingly  large  amounts  of 
bandwidth  from  the  networking  medium. 

DERA  the  team  leader  of  the  Combat 
System  Integration  (CSI)  programme, 
(which  is  a  five-year  UK  Ministry  of 
Defence  research  programme,  integrated 
with  the  Defence  Procurement  Agency 
(DPA)  and  UK  industry)  has  many  areas  of 
research,  one  of  which  is  High  Capacity 
Networks. 

There  are  currently  several  avenues  of 
research  into  LANs  and  Wide  Area 
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5.2  Future  Systems  -  Interoperability 

We  are  now  looking  at  more  than  just 
integration  of  a  system  on  a  platform. 
Interoperability  between  different  platforms 
at  force,  joint,  and  coalition  level  is 
required.  DERA  is  currently  working  on 
research  programmes  to  bring  together 
equipments  in  to  the  same  virtual 
battlespace  to  aid  future  interoperability 
studies. 


Currently  DERA  LBTS  have  a  DIS/HLA 
gateway  fitted,  which  enables  data  link 
trials  to  use  training  equipment 
manufactured  by  a  US  Company,  linked  to 
the  SIF,  and  via  land  lines  to  ships  in  port. 


The  building  of  synthetic  environments  to  Figure  3  DERA  CSI  Testbed 

meet  both  testing  and  training 

requirements  has  led  to  a  number  of 

standards  being  developed  world  wide  to 

support  the  interoperability  of  simulation 

systems.  Distributed  Interactive  Simulation 

(DIS),  and  the  future  development  of  a 

new  standard  High  Level  Architecture 

(HLA),  are  examples  of  architecture 

standards  that  are  intended  to  facilitate 

interoperability  between  all  levels  of 

simulation.  DERA  have  developed  the  CSI 

Testbed,  Figure  3.  This  is  a  generic 

combat  systems  integration  facility  where 

many  demonstration  or  real  equipments 

can  be  integrated  and  tested  in  a 

comprehensive  synthetic  environment.  It  is 

being  used  to  link  to  other  such  testbeds  in 

the  UK  and  US.  It  makes  extensive  use  of 

both  DIS  and  HLA  standards  for  the 

simulators  and  stimulators. 
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6  The  National  Maritime  Reference  Centre 

DERA  have  recently  taken  over  ownership 
and  operation  of  the  Land-Based  Test 
Site.  DERA  have  already  relocated  the 
Combat  System  Integration  (CSl)  research 
team  to  the  LBTS.  This  initial  extension  of 
capability  at  the  LBTS  will  form  the 
nucleus  of  a  National  Maritime  Reference 

The  NMRC’s  purpose  is  to  reduce  risk  at 
all  stages  of  the  acquisition  and  ownership 
process  and  be  driven  by  a  need  to  ensure 
that  operational  capability  is  maintained 
throughout  a  platform’s  life.  Its  aim  is  to  be 
recognised  as  an  authority  for  high  level 
systems  integration,  performance 
evaluation,  support  and  training.  The 
NMRC  should  accommodate  industry 
interests  and  initiatives  which  maybe 
offered  as  part  of  a  shared  approach  as 
well  as  other  DERA  initiatives  like  the 
Virtual  range. 


Figure  4  Wide  Area  Integration  Facility 

The  virtual  range  concept  would  enable 
ships  to  carry  out  trials  with  more  than  just 
real  threats  from  the  range  but  would 
include  virtual  tracks.  This  would  also  test 
the  system  under  a  higher  load  and  put 
the  operators  in  a  more  realistic 
environment,  making  best  use  of  the 
range. 

To  fully  realise  the  NMRC  requires  an 
approach  which  satisfies  the  aspirations  of 
stakeholders.  Such  an  approach  needs 
the  support  and  active  involvement  of  both 
MoD  and  industry,  which  in  turn  means  it 
has  to  be  pragmatic  and  cost  effective.  In 
turn  this  suggests  the  NMRC  should  be 
shared  and  focus  on  developing  core  skills 


and  capabilities  through  the  realisation  of 
specific  programmes  in  a  incremental  step 
by  step  manner.  Links  with  other  reference 
sites  such  as  Land  Reference  Centre  at 
Blandford,  are  envisaged,  as  well  as  the 
established  links  to  US  facilities. 


7.A  Final  Thought 

The  facilities  at  Portsdown  have  now  been 
a  landmark  on  the  Portsmouth  sky  line  for 
some  fourteen  years  and  will  continue  to 
evolve.  During  this  period  the  UK  National 
Audit  Office  carried  out  a  study  in  to  the 
Type  23  Frigate  Programme,  this  reported 
that  potential  savings  for  the  Type  23 
Project  over  a  25-30  year  period  based  on 
records  from  activities  at  the  SIF  was  in 
the  order  of  6  ship  years. 

Looking  to  the  future,  the  site  is 
evolving  as  a  node  of  a  global  network  of 
reference  centres  demonstrating  platform, 
force,  joint  and  coalition  interoperability,  of 
which  this  will  be  the  National  Maritime 
Reference  Centre . 


©  Crown  Copyright  2000  DERA.  Published 
with  the  permission  of  the  Defence 
Evaluation  and  Research  Agency  on 
behalf  of  the  Controller  HMSO. 
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Abstract 

This  paper  takes  time  out  to  consider  the 
fundamentals  involved  in  the  engineering  of  large 
systems.  It  identifies  weaknesses  in  the  current 
systems  engineering  and  test  and  evaluation 
practices.  It  begins  with  development  of  a 
pragmatic,  model  of  the  Systems  Engineering 
problem  space  in  which  many  design  teams  are 
interacting  with  the  common  goal  of  delivering  a 
system  on  time,  at  least  cost  and  to  at  least 
expected  performance.  This  model  is  then  used 
to  reveal  fundamental  deficiencies  of  current 
methodologies  showing  that  T&E  and  SE  are 
essential  partners  in  the  whole  life  cycle  problem. 

It  also  exposes  where  the  serious  barriers  arise 
to  more  effective  systems  development. 

Introductory  Remarks 

In  all  endeavours  it  is  necessary  to  keep  track  of 
the  fundamentals  involved  and  to  study  their 
changes.  This  can  reveal  new  insights  in  how  to 
bring  about  improvements.  This  paper  addresses 
the  practices  of  Test  and  Evaluation  (T&E),  and  its 
highly  related  Systems  Engineering  (SE),  in  the 
context  of  them  supporting  the  whole  of  life  design 
and  operation  of  large  complex  systems. 

For  too  long  these  two  areas  have  been  followed 
with  less  than  adequate  interaction.  For  example 
the  T&E  oversight  activities  of  the  US  DoD  Office 
of  Secretary  of  Defense  (OSD)  had  little 
representation  of  Systems  Engineering  until  recent 
times;  the  International  Association  for  Test  and 
Evaluation  (ITEA)  conference  content  can  be  said 
to  cover  SE  in  context  but  presentations  rarely 
address  T&E  practice  in  its  relationship  to  that  of 
SE.  Even  more  surprisingly  is  the  fact  that  the 
content  of  symposia  and  topics  of  the  37  Technical 
Committees  of  the  International  Council  of 
Systems  Engineering  (INCOSE)  give  only  scant 
attention  to  T&E.  Leading  SE  text  books  also  do 
not  give  adequate  coverage  to  T&E.  The  most 
recent  major  statement  of  the  knowledge  of  SE  is 
Sage  (1999).  It  has  little  of  relevance  about  T&E 
and  how  it  should  be  implemented  as  a  risk 
reduction  process  in  SE  practice. 


It  is  reasonable  to  assert  that  the  prime  drivers  in 
current  engineering  of  defence  and  aerospace 
systems  seek:- 

•  Improved  commercial  gain 

•  Improved  defence  capability 

•  Improvement  in  all  of  the  cost,  time  and 
performance  factors 

•  National  accountability  and  reputation 

The  key  area  of  concern  here  is  the  process  by 
which  systems  are  engineered,  this  then  largely 
deciding  how  T&E  is  best  implemented.  These  two 
factors  cannot  be  treated  as  independent  issues. 

The  three  main  risk  areas  in  SE  are  the  ‘CTP’ 
factors 

•  T  -  time  to  completion  (controlled 
purposefully  by  project  management) 

•  C  -  cost  to  completion  (controlled 
purposefully  by  accounting  processes) 

and,  with  these  two  met,  the  system  also  has 

•  P  -  whole  of  life  technical  performance 
needed  (controlled,  or  should  be,  by 
purposeful  and  appropriate  T&E) 

To  perform  these  three  risk  reduction  strategies 
simultaneously  requires  they  all  be  well  integrated. 
T  and  C  practices  are  but  T&E  is  not,  as  will 
developed  later. 

Where  would  one  look  for  the  'Best  Practice’  for 
this  combination?  In  the  1960-70s  defence  and 
commercial  systems  were  developed  as  though 
they  are  quite  different  worlds.  Today  they  have 
come  together  but  there  is  clear  evidence  that  they 
are  not  fully  the  same  and  thus  the  two  practices 
can  only  converge  so  far  -  and  we  appear  to  be 
near  that  limit  today. 

In  defence  systems  the  leadership  in  their 
engineering  is  arguably  with  the  US  defence 
contractor  sector  with  the  UK  MoD  having  just 
picked  up  the  leading  features  via  the  Smart 
Procurement  Initiative. 

In  the  commercial  sector  it  the  multi-national  Level 
1  systems  houses  that  are  leading  the  way. 
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Defence  procurement  is  fast  giving  way  to 
emulating,  where  it  can,  the  commercial  practices. 

The  last  decade,  especially,  hias  seen  major 
changes  in  SE  practice,  with  T&E  being  pulled  on 
to  a  mixed  extent. 

One  would  be  forgiven  for  believing  that  some 
major  new  ideas  have  been  implemented. 
However,  analysis  of  what  has  been  happening 
shows  they  have  been  a  continuum  of 
extrapolated  method.  The  boldness  to  be  noted  is 
the  extent  to  which  systems  houses  have  changed 
their  practices  through  large  cultural  shifts  and 
heavy  facilities  and  support  tools  investments. 

This  brief  overview  has  set  the  background  for 
investigating  where  major  improvements  are 
needed. 

Trends  in  Systems  Engineering 

Since  the  1940’s  systems  development  has 
needed  formalised  processes  to  be  able  to 
manage  its  ever-  increasing  complexity.  The  US 
has  generally  taken  the  lead  with  standardisation 
of  the  SE  process  but  other  less  well  known 
practices  have  tracked  this  culture.  All  recognise 
the  essential  need  for  serial  stages/phases  - 
Concept,  Realisation.  Detailed  Design, 
Manufacture,  Operation,  Disposal. 

SE  Process  adoption  began  as  a  formalised 
expression  with  the  Mil-Std  499  of  1969.  This  has 
become  known  as  the  Waterfall  Process.  Here  the 
philosophy  is  that  each  phase  is  a  self-contained 
activity  for  which  output  products  are  the  material 
needed  to  feed  the  next  phase.  A  good  descriptor 
is  that  documents  are  Thrown  over  the  Wall’  to 
the  next  phase. 

This  process  was  predicated  on  the  belief  system 
that  Customer  Requirements  could  be  set  early, 
that  they  would  be  correct,  that  they  could  be 
reticulated  ever  on  downward,  and  that  the  sum  of 
the  parts  when  put  together  would  give  the 
performance  needed.  It  lacked  enough  feedback 
so  workers  went  the  way  they  saw  was  best  with 
insufficient  regard  as  to  whether  their  contribution 
was  ‘right’.  T&E  was  present  as  localised  designer 
testing  and  milestone  evaluations  for  payments. 
There  was  little  emphasis  on  whole  of  project  life 
T&E  planning.  Waterfall  was  too  slow,  too 
expensive  and  did  not  produce  as  well  as 
expected.  This  was,  however,  a  major  step 
forward  for  the  process  was  externalised,  very  well 
documented  and  today  has  given  us  a  wealth  of 
information  on  ‘what  to  do’  with  some  'how  to  do’ 
information  on  managing  large  SE  processes. 


From  this  base  line  process  evolution  has  been 
taking  place  that  leads  us  to  where  we  are  today. 
The  first  major  step  fonward  was  to  recognise  that 
the  link  of  events  had  two  kinds  of  activity.  First, 
paper  studies.  Second,  manufacture  of  objects. 
(Early  SE  processes  used  in  defence  systems 
gave  little  attention  to  ‘whole  of  life’  issues.) 
These  two  could  be  folded  into  a  Vee  allowing 
strategic  test  and  evaluation  to  be  applied.  This 
improvement  allowed  the  various  steps  of  the 
implemented  design  to  be  progressively  tested 
against  the  breakdown  requirement.  The  V 
diagram  is  used  extensively  today  and  is  well 
documented.  Its  T&E  is  more  efficient  but  it  also 
lacks  a  sound  forward-looking  T&E  plan  that  is 
able  to  give  technical  performance  risk 
management  guidance. 

The  UK  MOD  Downey  cycle  is  an  adaptation  of 
these  principles. 

The  demands  for  better  ‘Time,  Cost,  Performance’ 
factors  pulled  on  investigation  of  how  to  improve 
the  SE  process  still  further.  In  the  1980’s 
academic  and  industrial  research  in  manufacturing 
systems  picked  up  on  the  notion  of  ‘Concurrent 
Engineering’  (CE).  This  is  a  simple  philosophy. 
Can  any  activities  of  later  phases  be  started 
before  the  phase  preceding  them  is  completed? 
Rather  than  wait  for  final  design  details  can  the 
next  steps  be  started,  even  if  some  later 
adjustment  is  needed?  This  cuts  down  the  time  of 
the  cycle  and  also  better  integrates  the 
experiences  of  later  phase  activity.  This  may  have 
been  a  bonus  from  CE  research  but  it  led  to  the 
integration  of  a  representation  of  all  facets  right 
through  whole  of  life.  This  implements  the  'early 
error  correction’  philosophy  wherein  it  is  well 
established  that  early  correction  can  yield 
significant  overall  savings.  By  1990  the  concept  of 
Integrated  Product  and  Process  Development 
(IPPD)  was  in  use  in  dominantly  US  industry.  By 
1992  such  corporations  as  Boeing  had 
implemented  CE  and  IPPD  in  a  major  way,  an 
early  well-known  product  from  this  new  culture  and 
rpethodology  being  the  Boeing  777  air  plane. 

Integrated  Products  Teams  (iPTs);  concurrent 
engineering;  guidance  SE  process  standards; 
process  re-engineering;  and  computer  supported 
SE  processes  are  the  ‘Best  Practice’  of  today. 

So  where  to  next?  Can  there  be  other 
philosophies  for  tuning  the  SE  process? 

There  are  indeed  a  number  of  variations  that 
evolve  the  process  still  further  but  they  tend  to  be 
special  cases  of  the  above.  They  are  all  used  to  a 
limited  extent.  They  include  incremental 
acquisition  in  which  the  first  phases  develop  a 
basic  design,  enhancing  the  system  performance 
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in  later  steps,  (This  is  the  process  preferred  in  the 
UK  MoD’s  Smart  Procurement.)  Another  is 
continuous  procurement  in  which  improvements 
are  added  as  small  increments.  Here  resources 
are  fed  in  to  gradually  build  up  capability  as  it  can 
be  afforded. 

Taking  the  various  SE  processes  to  the  limit  of 
where  they  might  be  expected  to  evolve  suggests 
the  end  point  would  be  a  small  grained  system  in 
which  concurrency  is  smoothly  applied.  This  is 
explained  later. 

The  latest  major  activity  to  pick  up  on  these  good 
features  in  national  way  is  the  Smart  Procurement 
Initiative  of  the  UK  MoD.  It  takes  on  board  the 
best  of  the  above  practices.  It  gives  a  great  deal  of 
flexibility  to  leadership  to  use  these  various 
philosophies. 

But  is  the  SE  process  well  enough  understood  to 
tailor  it?  Are  there  aspects  that  can  be  improved 
in  as  powerful  a  manner  as  did  CE  and  IPPD 
philosophies?  This  will  be  addressed  below. 

Here  has  been  summarised  50  years  of  SE 
process  development.  An  overview  is  now  needed 
for  T&E. 

Trends  in  Test  &  Evaluation 

All  well  managed  projects  have  considerable  test 
content  in  them.  Requirements  have  tests 
associated  with  them.  The  high  level  ‘purposeful’ 
and  the  ‘suitability’  (R&M  -  Reliability  and 
Maintenance  -  tends  to  be  the  term  used  in  the  UK 
MOD)  statements  define  the  system  issues  that 
are  to  be  satisfied  to  meet  the  needs  of  the 
customer  and  users.  These  statements  are  the 
logical  starting  point  for  setting  up  tests  that 
provide  data  to  feed  evaluations. 

As  a  design  develops  the  design  team  conducts 
localised  tests  to  be  sure  the  design  meets  the 
requirements  that  have  flowed  down  to  them  by 
the  requirements  breakdown  process. 

This  kind  of  evaluation  is  common  practice  and  is 
often  not  stated  as  a  formal  process  but  one  being 
conducted  within  the  team’s  normal  practices.  This 
is  appropriately  called  ‘informal  T&E’.  The 
practitioners  feel  confident  they  know  what  to  do  - 
but  usually  cannot  justify  why  they  tested  what 
they  have.  Their  professionalism  is  thought  to  be 
sufficient  basis  for  deciding  T&E  issues. 

Despite  all  the  good  intentions  there  is  much 
evidence  to  show  that  adding  up  the  sum  of 
reticulated  requirements  does  not  lead  to  a 
satisfactory  system.  A  simple  reason  is  that  the 
lower  parts  too  often  are  not  interfaced  to  higher 


and  other  levels  sufficiently  well  to  know  what  the 
sub-system  they  are  building  is  supposed  to  be 
able  to  do. 

Around  1 980  key  US  sponsors  became  tired  of  too 
many  failed  defence  systems  -  the  CTP  factors 
were  variously  not  met.  A  major  study  led  to 
Congress  passing  laws  requiring  all  major  defence 
projects  to  have  much  improved  T&E  processes 
and  that  there  be  oversight  by  Pentagon 
government  of  programme  performance.  This  led 
to  the  creation  of  the  separate  OSD  offices  for 
development  and  for  operational  T&E.  Major 
initiatives  included  requirement  for  the  Test  and 
Evaluation  Master  Plan  (TEMP),  certification  of 
T&E  employees,  standards  for  T&E  practice, 
budgets  for  national  T&E  improvement  programs 
and  central  coordination  of  many  T&E  services. 
The  TEMP  was  primarily  created  for  programs  to 
give  OSD  staff  oversight  of  crucial  information 
needed  to  be  able  to  monitor  programme 
progress. 

Political  reasons  appear  to  have  separated 
development  T&E  oversight  from  that  for 
operational  T&E.  The  US  Service  arms  continue  to 
conduct  their  own  test  facilities  and  T&E 
processes  in  the  manner  that  they  choose.  The 
TEMP  is  provided  to  the  OSD  as  an  early  part  of 
Program  documentation. 

In  1999  a  US  DoD  national  review  recommended 
a  much  closer  link  between  whole  of  life  issues, 
SE  practice  and  the  OSD  T&E  agencies. 
Reorganisation  recently  took  place  to  this  effect 
but  still  with  DT&E  being  a  separate  OSD  activity 
to  OT&E. 

A  TEMP  recognises  that  informal  T&E  is  essential 
and  that  a  forward  looking  T&E  plan  is  also 
needed  that  ties  the  data  obtained  from  tests  back 
to  the  Critical  Issues  (CIs)  that  are  set  into  the 
Customer  high  level  needs  statements.  This  helps 
to  ensure:- 

•  The  right  tests  and  evaluations  are  made 

•  Test  requirements  are  planned 

•  Facilities  are  available  when  needed 

•  Test  resources  are  used  efficiently 

•  Test  effort  is  traceable  to  the  essential 
needs. 

Today  TEMP  documents  are  relatively  common¬ 
place  in  US  defence  programs  -  but  seemingly 
because  of  legal  requirements  as  much  as  any 
reason!  They  are  not  always  used  to  good  effect 
as  it  seems  the  virtues  of  such  are  not  well 
recognised  with  T&E  being  allowed  to  fall  back  to 
the  final  'see  if  it  is  to  the  need’  situation. 
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TEMPS  are  not  just  used  as  a  defence  systems 
practice;  some  US  industries  use  them. 

For  some  years  the  Australian  DoD  has  called  for 
TEMPs  as  part  of  Defence  Instructions  re  program 
management  but  still  few  are  actually  written.  It 
appears  that  their  risk  control  value  is  not 
appreciated. 

In  the  UK  some  contracts  to  contractors  have 
called  for  TEMPs  but  it  has  not  been  a  common 
practice  -  as  far  as  can  be  seen  for  contractors  are 
not  that  forthcoming  on  how  they  conduct  T&E 
within  programs. 

In  recent  times  the  various  western  defence 
acquisition  agencies  have  recognised  the  lead 
possessed  by  the  large  aerospace  and  defence 
companies  and  are  moving  their  cultures  to  make 
the  best  of  that  experience.  Whilst  there  appear  to 
be  major  gains  to  be  had  there  are  also  clear 
reasons  why  the  defence  business  appears  to 
never  be  able  to  make  use  of  all  of  their 
advantages. 

This  short  overview  also  needs  to  include  the 
impact  of  SE  and  T&E  practices  as  used  in  other 
areas  of  large  commercial  practice.  One  dominant 
area  is  the  telecommunications  field.  This  has  a 
total  turnover  much  greater  than  defence 
spending;  regards  a  $billion  programme  as  a 
normal  lead-in  cost;  works  within  very  rigid 
interfacing  protocols;  implements  comparatively 
small  advances  to  a  proven  system;  uses  proven 
technology  and  has  to  upgrade  their  systems  in  a 
seamless  manner  over  24hrs  a  day,  all  year,  all 
time.  For  all  of  this  it  also  has  huge  commercial 
failures. 

The  car  industry  is  also  after  major  improvements 
in  the  CTP  factors  in  order  to  stay  competitive  in 
an  over  supply  situation.  It  makes,  compared  with 
defence  needs,  only  small  technical  changes  to  its 
deliverables.  Its  product  is  very  mature  and  there 
is  little  room  to  move  in  design.  Its  also  works 
within  a  tightly  standardised  interfacing  situation. 

There  are  certainly  things  to  learn  from  the  civilian 
systems  world.  The  US  DoD  has  recently  had  a 
three  person  Military  Fellows  team  study  the 
situation.  The  major  aerospace  companies  have 
interesting  initiatives  in  place  to  further  tune  the 
CTP  factors.  Examples  include  the  Boeing  ‘Create 
New  Air  Plane’  initiative;  the  Lockheed  ‘Lean  Air 
Craft’  initiative;  and  the  Jet  Propulsion  Laboratory 
‘Create  New  Products’  initiative.  There  do  not 
seem  to  exist  counterpart  programmes  in  the  EU! 

So  why  give  this  background?  This  paper  now  is 
in  the  position  to  explore  future  potential  trends 
suggesting  how  and  where  SE  and  T&E  will  be 


able  to  still  obtain  major  improvements  in  the 
chase  for  optimum  CTP  factors. 

The  preceding  material  lets  us  approach  two  main 
issues,  one  in  SE  and  the  other  in  T&E,  where 
considerable  improvement  can  still  be  obtained. 
These  are  now  developed. 

The  Teaming  Model  of  Systems  Engineerinq 

The  general  development  of  SE  practice  in 
Defence  industries  appears  to  concentrate  on 
tuning  of  the  SE  process  itself  The  SE  process 
standards  started  with  heavy  ‘prescription’  and 
have  evolved  now  to  be  ‘guidelines’  that  leave 
considerable  flexibility  of  implementation  up  to  the 
project  manager  and  systems  creator. 

The  latest  trends  are  to  apply  ‘model-based’ 
thinking  to  the  SE  activity.  This  trend  is  relatively 
new  in  its  acceptance  for  study.  It  is  being 
interpreted  to  mean  anything  ranging  from:- 

•  Convert  the  SE  process  guidelines  from 
words  into  interacting  block  diagrams  (first 
step  in  implementation  of  what  the  control 
systems  discipline  calls  ‘systems 
identification’) 

to 

•  Develop  a  formal  mathematical  model  that 
can  be  executed  in  a  computer  to  gain 
optimal  performance  (the  later  steps  in 
Systems  identification) 

Model  based  thinking  is  standard  practice  in 
engineering  of  those  kinds  of  physical  systems 
that  succumb  easily  to  formal  ‘hard’  mathematical 
description. 

To  this  end  INCOSE  has  established  a  strongly 
supported  Working  Group  for  Model  Based  SE.  It 
held  a  major  session  at  its  annual  Symposium 
held  in  Brighton  UK  in  June  1999. 

It  is  logical  to  see  this  sort  of  development  emerge 
for  it  is  driven  much  by  engineers  who  are  steeped 
in  the  reductionist  culture  and  have  little  time  for 
the  alternative  phenomological  way  of  thinking  that 
is  found  in  the  humanities  -  where  it  is  accepted 
that  formal  models  of  people  systems  are  not 
feasible  to  any  really  useful  extent. 

Now  presented  is  a  simple  ‘show  stopper’ 
argument  that  sheds  light  on  the  futility  of  chasing 
model-based  SE  as  the  main  thrust  for  gaining 
improvements  of  significant  order. 

Discourses  in  SE  use  the  SE  process  as  the 
foundation  -  define  the  process  phases  and  then 
hang  on  that  framework  the  knowledge  relevant  to 
the  parts  of  that  framework. 
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The  human  organizational  issues  involved  are 
given  low  exposure  in  SE  organizations.  So  let’s 
start  with  a  different  foundational  model  being  that 
the  task  of  a  SE  activity  is  to  integrate  the 
contributions  of  hundreds  of  engineers  to  all  seek 
to  maximise  the  way  they  add  value  by  enhancing 
a  given  or  new  system  toward  new  levels  of 
performance.  This  is  here  called  the  Pragmatic 
Teaming  (PT)  model. 

Figure  1  shows  the  first  stage  of  its  development. 
The  key  features  are:- 

•  Teams  (small  oval  cells)  exist  to  design  a 
system  that  satisfies  a  Needs  Statement 
giving  the  best  performance  for  least  cost 
and  cycle  time 

•  They  ideally  work  together  exchanging 

their  existing  and  developing  knowledge  to 
move  the  whole  design  effort  toward 

delivery  of  the  required  product  in  a 
continuous  manner 

•  Below  the  overall  planning  and  leadership 
level  they  work  on  tasks  within  their  area 
of  competency  on  the  ‘already  knowing 
what  to  do’  basis. 

•  All  teams  carry  out  much  the  same 

intellectual  task  -  identified  as  the  SEP 
(System  Engineering  Process) 

•  Interface  studies  (n^  matrices/diagrams 

methods)  show  teams  need  to  interface 
the  knowledge  of  their  cells  with  at  least 
50%  of  that  of  all  other  cells. 

•  Strategic  grouping  reduces  the  interfacing 
problem  somewhat  but  still  leaves  a 
massive  knowledge  flow  situation. 


Output  Product  to  Customer  Need 


Figure  1.  First  Step  toward  Pragmatic  Teaming 
Model 


Overall  this  model  (called  the  ‘organismic’  model 
in  organisational  science  for  the  parts,  the  organs, 
are  all  doing  their  own  thing)  is  not  successful  as  it 
lacks  overall  guidance.  It  identifies  well  with  the 
waterfall  situation  in  which  the  cells  carry  out  their 
work  to  the  reticulated  specifications  with  less  than 
enough  communication. 

Pragmatics  of  how  a  system  is  developed  are  now 
used  to  further  develop  this  first  stage  model  by 
adding  leadership  to  each  of  the  teams.  This 
allows  communication  of  knowledge  between 
teams  and  gives  the  whole  overall  leadership  -  the 
Systems  Engineer  function. 

Fig  2  gives  the  pragmatic  model  as  SE  is  generally 
practiced  today.  This  model  is  known  as  the 
‘neuro-cybernetic’  organisational  model  (or  ‘viable’ 
model).  It  is  well  established  in  systems  science 
and  is  not  contentious  as  to  representing  the  SE 
activity  reasonably  well. 

With  this  simple  model  we  can  easily  and  soundly 
identify  the  problem  that  appears  to  this  writer  to 
be  major  showstopper  to  improvement  in  SE 
activity. 
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Figure  2.  Pragmatic  Model  Able  to  Show 
Barrier  to  Progress 

Let  us  recall  that  the  mission  of  the  design  team 
system  is  to  move  the  many  teams  involved 
toward  the  ‘right’  end  result  with  the  CTP  factors 
as  optimisation  requirements. 

The  ‘engineering  science’  methodology  for  finding 
the  optimal  solution  to  this  kind  of  network  problem 
is  to  treat  this  set  of  interacting  teams  as  a 
topology  calculation  problem.  The  first  step  is  to 
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set  up  descriptive  equations  for  the  behaviour 
within  a  team  in  terms  of  input/output 
relationships.  Then  are  established  the  linkages 
between  all  blocks  and  the  behaviour  of  those 
linkages  as  additional  processing  blocks.  With  this 
all  done  it  is  then  feasible  to  set  in  initial  conditions 
and  relax  the  system  toward  its  optimal  state. 
Model-based  systems  engineering  appears  to  be 
advancing  according  to  this  paradigm. 

This  methodology,  however,  will  soon  come  upon 
a  great  barrier  to  its  effective  deployment. 

If  the  flux  within  the  network  consisted  as  only 
mass  or  energy  flows  then  descriptive  analytical 
formal  methods  promise  limited  optimisation 
success  -  even  for  non-linear  situations.  However, 
the  stuff  flowing  between  cells  is  ‘information’  and 
‘knowledge’  as  well  as  ‘hard  data’. 

At  the  current  state  of  understanding  we  are 
nowhere  as  advanced  with  formal  characterisation 
of  human  messaging  as  we  are  with  formalised 
mathematical  calculation  of  the  properties  of 
networks.  Engineers  have  little  understanding  of 
the  scholarship  of  human 

communication  -  such  as  of  ‘semiotics’  (the  theory 
of  messaging);  Knowledge  Based  Systems  and 
linguistics  and  grammatics;  all  being  disciplines 
where  we  might  find  a  foundation  for  treating 
knowledge  flows  within  nodal  networks  of 
cooperating  teams. 

If  we  cannot  reduce  human  communications  to 
format  formulae  representation  then  we  are  a  long 
way  from  being  able  to  optimise  teams  using 
‘engineering’  control  systems  paradigms. 

At  present  considerable  effort  is  being  placed  in 
standardising  the  SE  ‘data  link’  between 
computers  used  by  designers  -  such  as  by  the 
SEDRES  project  and  its  developing  standard.  This 
writer  suggests  the  data  link  aspect  is  a  much 
lesser  issue  when  compared  with  the  need  to 
adequately  understand  the  human  communication 
need.  Both  types  of  interfacing  links  must  be  well 
understood  in  order  to  make  a  significant  impact 
on  optimisation  of  SE  activity. 

We  need  considerably  more  research  in  place  to 
investigate  the  human  teaming  optimisation  model 
and  its  knowledge  flows.  There,  however,  exists 
only  a  handful  of  researchers  in  the  world  looking 
at  this  highly  important  aspect.  One  line  of  attack 
is  ‘Soft  Systems’  as  constituted  at  present 
(originated  by  Checkland  (1981)  and  extended  by 
Flood  and  Jackson  (1991)).  Progress  there  is  little 
more  than  past  identifying  the  problem  in 
organisational  terms.  For  progress  to  be  made 
engineers  and  scientists  must  cooperate  more 


with  organisational  scientists,  with  cognitive 
psychologists  and  other  humanities  disciplines. 
The  answers  do  not  appear  to  lie  in  the 
scholarship  or  established  practices  of 
engineering! 

In  summary,  SE  process  is  the  current  emphasis 
of  improvements  to  SE  practice.  However,  by 
looking  at  a  Pragmatic  Teaming  model  of  the  SE 
activity  it  is  easy  to  see  and  prove  that  a  major 
hold-up  to  progress  is  lack  of  attention  being  given 
to  the  human  communication-interfacing  problem 
of  knowledge,  as  well  as  data,  within  a  teaming 
network.  This  is  a  vital  area  in  need  of  greatly 
improved  understanding  in  terms  of  computer 
network  operations.  Programmes  such  as  the 
major  SEDRES  one  may  well  improve  the 
interfacing  for  data  links  between  designers  but 
the  meaningful  knowledge  aspect  will  still  remain  - 
and  it  is  tough  research  problem! 

Attention  has  been  given  to  the  SE  process  -  yet 
this  is  a  T&E  meeting.  Why? 

T&E  is  the  process  by  which  technical 
performance  risk  can  be  better  controlled.  Its 
effective  application  requires  a  deep  appreciation 
of  the  system  operation  to  which  it  is  being 
applied.  T&E  practitioners  rarely  have  experience 
in  the  SE  aspect,  and  vice  versa.  This  short 
summary  might  assist  bridge  that  gap  for  they 
have  not  been  adequately  interfaced  to  date. 

Let  us  now  look  into  where  T&E  practice  can  be 
improved  to  assist  optimise  the  CTP  optimisation 
factors. 

T&E  Practice  Based  on  Foundational  Concepts 

Implementation  of  T&E  can  be  built  up  from  sound 
scientific  foundations.  This  has  been  recognised  in 
the  US  DoD  T&E  process  as  defined  in  their 
TEMP  guidelines  and  later  teaching  courses.  That 
approach  has  been  further  developed  by  work  at 
the  Australian  Centre  for  T&E  (now  part  of  the 
Systems  Evaluation  and  Engineering  Centre)  and 
more  recently  in  the  teaching  of  the  new  MSc  in 
Systems  T&E  at  the  University  College  London. 
Three  key  processes  can  be  identified  as  essential 
to  implementation  of  technical  risk  control.  These 
are  a  use  of  a  defined  Systems  Engineering 
process,  a  T&E  process  and  many  Measures 
Trees. 

An  adequate  account  of  the  nature  and  scope  of 
SE  processes  used  has  already  been  given. 
Sound  T&E  cannot  be  applied  unless  the  process 
is  understood  and  documented. 

The  T&E  process  can  be  identified  as  the 
procedure  for  establishing  new  knowledge  about 
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the  performance  of  parts  and  the  whole  of  a 
system  design  and  operation  activity.  It  has  been 
well  established  that  this  is-  consistent  with  the 
steps  of  the  well-accepted  'Scientific  Process’ 
used  to  gather  new  knowledge  of  a  dynamically 
changing  activity.  Fig  3  shows  a  summary  chart 
comparing  the  scientific  process  with  the  steps  of 
the  T&E  process. 


Scientific  Method 

T&E  Process 

I.  Develop  Hypothesis 

1 .  Identify  Question/Problem 

1.  Develop  Test  Objectives 

2.  Formulate  Hypothesis 

2.  Estimate  Performance 

II.  Experiment 

3.  Plan  Experiment 

3.  Develop  Test  Method 

4.  Conduct  the  Experiment 

4.  Collect  Test  data 

5.  Analyse  Results 

5.  Calculate  MOPs 

III.  Verify'  Hypothesis 

6.  Check  Hypothesis 

6.  Compare  with  Thresholds 

7.  Refine  Understanding 

7.  Retest  or  extrapolate 

Fig  3  Chart  comparing  Scientific  Processes 

and  T&E  (From  SEEC  (1 998)) 

Sound  adherence  to  the  above  steps  of  the  T&E 
process  leads  to  reliable  and  relevant  knowledge 
gathering  about  the  performance  of  a  programme. 

However,  use  of  a  sound  SE  process  and  T&E 
process  is  not  sufficient. 

It  is  not  feasible  to  test  and  evaluate  every  part  of 
a  major  program  as  it  develops.  The  need  for 
designers  is  to  decide  what  are  the  critical  high- 
level  issues  and  set  up  the  T&E  Master  Plan  to 
ensure  these  are  addressed  and  monitored  as  the 
programme  progresses.  This  is  not  to  say  that 
T&E  is  not  needed  at  the  localised  level  but  that 
an  overlay  T&E  programme  is  needed  that 
addresses  the  overall  integration  issues 
adequately. 

For  each  Critical  Issue  (Cl),  identified  from  the 
high  level  capability  requirements  statements, 
there  needs  to  be  set  up  a  breakdown  of  several 
kinds  of  Measure.  This  breakdown  has  to  have  a 
traceable  hierarchy.  Terminology  used  for  the 
various  measures  is  taken  from  US  practices.  Fig 
4  shows  the  various  measures  in  the  generic 
measures  tree  (Some  US  DoD  teachers  call  this  a 
‘dendritic  diagram’  and  draw  the  tree  as  a 
horizontal  form.) 

To  establish  the  overall  programme  capability  at 
any  time  the  current  value  of  all  measures 
predicted  to  begin  with  are  calculated  according  to 
the  tree  trace  with  weightings  applied  to  each.  This 
gives  a  current  estimate  of  the  top  level  need  but 
cannot  yield  its  level  of  uncertainty.  This  is 


obtained  by  assigning  the  best  estimate  of 
uncertainty  to  each  measure  at  the  time.  With 
these  in  place  it  is  then  relatively  straightforward  to 
calculate  the  uncertainty  of  programme  maturity  at 
any  time.  These  calculations  can  largely  be 
automated  using  well-established  consensus  and 
statistical  methods  for  carrying  out  many  to  one 
mappings. 


Purposeful  Statement  with  Uncertainty  Limits 


<r  No  of  determinations  -> 

Fig  4.  Generic  Measures  of  the  Measures  Tree 

As  the  various  measures  become  better  defined 
the  overall  uncertainty  will  improve.  Examination  of 
selected  ‘highlighted’  measures  will  give  IPT 
leaders  and  Capability  Managers  the  crucial  data 
they  need  to  track  maturity. 

Sound  logical  considerations  suggest  these  three 
concepts  are  all  present  in  the  same  system  so 
they  must  be  capable  of  being  mapped  together  to 
yield  technical  performance  control  information. 

Having  done  that  the  TEMP  goes  a  way  toward 
defining  how  these  three  activities  are  integrated. 
However,  being  prepared  for  high  level  oversight 
study  of  several  TEMP  examples  suggests  there  is 
insufficient  traceability  for  the  various  measures  to 
mapped  back  up  the  trees.  Remembering  that  the 
only  ‘hard’  data  available  comes  from 
determination  of  the  Technical  Performance 
Parameters  (TPPs)  (with  some  more  subjective 
numbers  from  the  System  Performance 
Parameters  (SPPs))  traceable  mapping  is 
essential  to  use  measures  to  track  the  maturation 
of  the  Critical  Issues  and  thus  closeness  to 
customer  needs.  Any  serious  lack  of  traceable 
calculations  leaved  much  in  doubt. 
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It  seems,  in  the  USDoD  T&E  practices  that  the 
creation  of  the  TPP  and  SPP  parts  of  the  T&E  plan 
are  carried  out  with  less  than  openly  obvious 
procedures.  Army  creates  a  Systenns  Evaluation 
Plan.  Navy  and  Air  Force  construct  the  test  plans 
using  less  formal  methods. 

A  small  group  of  DERA  Ranges  Sector  staff 
enrolled  in  the  MSc  in  Systems  T&E  at  UCL  are 
currently  investigating  improved  integration  of 
these  key  processes.  Whilst  their  work  is  still  in  its 
early  stages  of  development  several  key 
conclusions  have  already  emerged  from  the 
assignment  work. 

•  The  SE  process  being  used  must  be 
identified 

•  The  T&E  process  must  be  used  to  develop 
the  essential  data  requirements  and  test 
plans 

•  Measures  Trees  must  be  set  up  from  the 
Critical  Issues  formed  from  customer 
requirements 

•  A  TEMP  document  must  be  prepared  that 
records  these  deliberations  and  having 
both  downward  and  upward  traceable 
mappings 

•  A  sound  explanation  of  the  downward  and 
later  upward  mappings  is  essential  to 
show  how  to  use  TPP  data  as  it  maps 
from  ‘hard’  quantitative  data  into  the  much 
more  qualitative  numbers  of  the  MOP, 

MOE  and  Cl  levels. 

A  most  significant  concept  developed  by  the  group 
is  the  need  to  establish  a  Programme  Critical 
Issues  database.  This  database  records  the  CIs, 
along  with  their  Measures  Trees  and  test  plans.  As 
each  phase  of  a  programme  is  developed  it  can 
also  yield  additional  CIs  of  its  own,  earlier  declared 
CIs  grow  in  maturity  and  eventually  outlive  their 
usefulness  to  become  the  source  of  knowledge  for 
other  programmes.  Fig  5  gives  an  impression  of 
the  T&E  data  flows  for  this  purpose.  This  aspect  is 
the  subject  of  current  study  as  this  appears  to  be  a 
novel,  or  at  least  extended,  concept  of  the  use  of 
centralised  metrics  databases. 

With  such  a  dynamic  T&E  database  the 
Programme  Manager  is  then  able,  at  any  time,  to 
see  how  the  various  CIs  are  maturing  by 
appropriate  use  of  the  measures  as  they  become 
available.  This  takes  the  well  established 
Technical  Performance  Measures  tracking  to  an 
improved  level  by  the  strict  addition  of  traceable 
and  scientifically  sound  creation  for  far  more 
issues  than  TPMs  usually  cover. 


Summary 

Two  salient  ways  to  improve  the  CTP  performance 
of  major  defence  systems  development  have  been 
presented. 

1.  Improved  appreciation  of  the  Knowledge  flow 
aspect  of  SE  activity  to  yield  better  team 
interfacing  would  significantly  reduce  design  error 
early  in  the  process  where  it  is  cheap  to  fix.  This 
is,  however,  hard  to  put  into  useful  practice 
because  of  the  embryonic  state  of  our 
understanding  of  the  problem  of  knowledge 
transfer  in  communications  networks. 

2.  More  rigorous  and  traceable  T&E  practices 
applied  to  well  defined  SE  process  implementation 
would  also  lead  to  early  error  detection  and 
control. 

This  can  be  implemented  now  as  its  features  are 
well  established.  It  will  increase  programme 
overhead  somewhat  but  can  confidently  be 
expected  to  save  those  costs  by  virtue  of  less 
costly  late  errors  arising. 

Well-implemented  T&E  high  level  planned 
processes  with  a  central  database  of  CIs  and 
Measures  Trees  have  the  potential  to  allow  a 
programme  to  proceed  through  its  phases  with  far 
less  ‘surprises’  making  metrics  available  that  can 
be  tracked  and  extrapolated  to  predict  toward 
sound  goals. 

To  do  this  well,  however,  needs  a  sound 
understanding  of  the  SE  process  and  this  in  turn 
makes  its  essential  that  T&E  and  SE  leadership 
staff  are  well  versed  in  both  areas  and  that  expert 
T&Eer’s  are  involved  very  early  in  programme 
phases. 
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Previous  Phases 


Future  Phases 


Fig  5.  Overview  of  Critical  Issues  Database  Flows  used  to  Assist  Control  Technical  Performance  of  a 

Programme.  (From  McDonald  (2000)) 
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Abstract 


The  changing  environment  in  world  economies  are 
creating  the  need  for  allied  governments  to  work 
together  to  develop  new  weapons  systems.  Under 
these  new  development  arrangements  evaluation 
costs  are  often  inflated  because  testing  is  repeated 
to  satisfy  the  standards  of  two  or  more  counties. 
The  United  States  and  the  United  Kingdom  are  the 
two  countries  most  extensively  involved  on  joint  or 
cooperative  development  programs,  however  the 
standard  models  for  operational  evaluation  vary 
greatly  between  the  US  and  UK.  US  MIL-STD 
operational  testing  and  UK  DEF-STAN  operational 
trials  ultimately  have  the  same  goal  of  providing  a 
complete  weapon  system  which  satisfies  end-user 
needs  while  being  affordable  and  maintainable.  At 
the  foundation  of  both  the  US  and  UK  models  is  the 
Concept  of  Operations  (ConOps)  for  the  weapon 
system  being  developed.  Beyond  this  point  the 
models  diverge.  The  US  model  provides  for  a  clear 
break  between  Developmental  Test,  performed  by 
the  contractor,  and  Operational  Test,  performed  by 
the  military.  It  also  requires  the  definition  of  strict 
Measures  of  Effectiveness  (MOEs)  and  Measures 
of  Performance  (MOPs).  The  UK  model  allows  for 
more  interactions  between  the  contractor  and  users 
during  development  and  Operational  Trials,  and 
defines  Trials  into  two  categories:  Single  Activities 
Tests  (SATs)  and  Multiple  Activities  Tests  (MATs). 
These  different  definitions  for  operational 
evaluation  are  more  alike  than  different  at  the  day- 
to-day  testing  level,  however  the  differences  in 
language  tend  to  create  duplicate  testing 
requirements  to  satisfy  each  country’s’  standards. 

This  paper  will  provide  a  basis  for  a  common 
operational  evaluation  language  by  mapping  the 
US  and  UK  definitions  and  objectives  to  each  other. 
Additionally  the  use  of  more  closely  integrated 
contractor/government  teams,  and  synergies 
created  by  combining  assets  and  understanding, 
will  be  presented  as  a  way  to  streamline  testing. 
This  will  create  a  foundation  for  a  new  model,  which 
will  satisfy  the  US  and  UK 


The  Challenge  of  Joint  Development 

We  finished  the  FAA  cert,  and  we  thought  we  were 
done... then  we  sold  the  plane  to  the  USAF and  the 
RAF  -  Anonymous 


Today’s  economy  is  becoming  increasingly  global 
and  with  it  so  is  the  defense  industry.  BAE  now 
has  an  American  and  Canadian  sector  just  as 
Lockheed  Martin  has  a  UK  company.  They  remain, 
however,  divided  by  national  policy  and  law  into 
separate  operating  companies  isolated  within 
specific  borders.  In  general,  no  other  multinational 
industrial  base  is  so  divided. 

Add  to  this  the  regulations  of  the  US  and  UK 
military  and  their  respective  procurement  policies, 
and  it  becomes  clear  that  any  company  seeking  to 
develop  and  market  a  new  military  aircraft  for  joint 
use  is  facing  many  challenges.  In  most  cases  in  the 
past  those  companies  haven’t  even  tried  to  save 
money,  they  just  did  the  testing  and  documentation 
twice,  once  in  the  US  and  once  in  the  UK. 

The  aerospace  industry  and  the  military  can  no 
longer  afford  this  extravagance  because  defense 
budgets  have  been  cut  to  their  lowest  levels  in 
decades.  Quality  aircraft,  which  meet  the  mission 
needs,  can,  and  should,  be  developed  jointly. 

The  US  DoD  and  UK  MoD  have  more  in  common  in 
terms  of  missions,  and  strategic  and  tactical 
objectives,  than  most  other  countries.  Both  the  US 
and  the  UK  have  good  and  bad  in  their  approach  to 
development  and  testing.  The  models  need  to 
brought  closer  and  the  best  practices  of  each  used 
to  facilitate  joint  programs,  and  do  it  once. ..right  the 
first  time. 

Defining  US  Operational  Test  and  Evaluation 

In  basic  terms  Operational  Test  and  Evaluation 
(OT&E)  is  concerned  with  determining  whether  the 
weapon  system  meets  the  needs  of  the  warfighter. 
OT&E  tests  to  concepts,  particularly  the  Concept  of 
Operations.  A  good  ConOps  includes  scenarios 


standards  for  operational  evaluation  and  produce 
more  effective  testing,  while  eliminating 
redundancy. 
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which  describe  the  basic  mission  of  the  system,  its 
operating  environment,  range,  payload,  crew, 
armament,  ground  support  equipment  (GSE), 
support  personnel,  maintenance,  deployability, 
mission  planning,  and  weaponeering.  This  provides 
the  planning  basis  for  OT&E. 


Figure  1  -  Typical  Concept  of  Operations 


By  US  law,  the  Operational  Test  organization  must 
be  independent  from  the  developing  agency  and 
contractors.  This  wall  in  the  past  has  created  a 
problem  with  “testing  in  a  vacuum”.  Often  the 
Operational  Test  organization  is  not  strongly 
involved  with  the  development  phase  and  has  not 
learned  all  of  the  nuances  of  the  system  by  the  time 
OT&E  begins.  Supporters  of  this  method  believe 
that  it  provides  a  more  fair  and  accurate  evaluation 
of  the  training  and  the  system,  while  detractors 
believe  the  time  could  be  used  more  effectively  if 
the  testers  knew  the  system.  Several  newer 
programs  have  begun  experimenting  with  bringing 
operational  testers  into  the  process  during 
developmental  testing.  These  experiments  have 
tended,  at  least  anecdotally,  to  field  more  mature, 
better-understood  weapon  systems. 

In  traditional  programs  where  OT&E  is  separated 
from  the  developers,  the  Operational  Test 
organization  begins  developing  an  Operational  Test 
and  Evaluation  Plan  by  defining  Critical  Operational 
Issues  (COIs).  These  are  derived  directly  from  the 
ConOps.  A  COl  should  address  significant  threats 
or  needs  for  the  new  weapon  system.  As  an 
example,  a  new  surveillance  and  command  aircraft 
may  have  a  COl  that  requires  it  to  track  enemy 
aircraft  at  300  miles  in  various  weather  conditions. 
The  COl  is  derived  from  the  need  to  keep  the 
aircraft  outside  of  hostile  airspace  where  it  can  be 
protected.  Another  example  of  a  COl  might  be  that 
a  cargo  plane  must  land  on  short,  unimproved 
landing  strips,  with  a  full  payload.  This  COl  allows 
for  the  aircraft  to  support  deployment  into  remote 
areas. 

OT&E  does  not  traditionally  test  “shalls”  from  the 
system  level,  or  lower  level,  specifications. 
Performance  criteria  may  be  derived  from 
development  specifications,  but  they  are  couched 


in  the  total  mission  environment,  not  taken  as  an 
absolute.  For  example,  a  subsystem  specification 
may  require  that  in  standard-day  conditions,  at  sea 
level,  an  engine  will  develop  80%  thrust  at  an  exact 
throttle  setting.  This  requirement,  tested  in  an 
engine  cell,  or  even  in  developmental  flight  testing 
under  controlled  conditions  can  be  verified  with 
great  certainty  and  accuracy.  In  an  operational 
environment,  however,  that  same  engine  may 
never  again  be  exposed  to  a  standard  day 
condition.  The  throttle  rigging  and  the  fuel  flow 
sensors  may  have  marginal  errors  induced  by 
maintenance,  all  of  which  will  add  up  to  a  slightly 
different  efficiency  for  the  engine  performance. 
Operational  test  in  this  example  would  determine 
whether  the  engine  efficiency  under  real-world 
conditions  was  within  acceptable  limits. 

OT&E  also  provides  a  valuable  period  where  the 
warfighter  gets  a  last  look  at  the  system  prior  to 
fielding  it.  This  can  translate  into  many  useful 
insights  that  can  be  added  to  flight  and 
maintenance  manuals,  and  training  materials.  Many 
times,  when  a  new  system  begins  OT&E,  the 
ConOps  is  refined.  This  occurs  because  inevitably 
the  system  will  provide  features  and  capabilities 
which  were  not  discussed  in  the  original  ConOps. 

Tvoicai  OT&E  Progression 

System  Assessment  OT&E  generally  begins  by 
evaluating  whether  the  system  works  as  advertised. 
Do  the  computers  boot?  Does  the  radar  run  and 
detect  targets?  Can  the  aircraft  carry  its  basic 
payload  over  the  required  range?  Many  of  these 
items  are,  in  the  US  model,  included  in  the 
Measures  of  Performance  (MOPs)  laid  out  in  the 
Operational  Test  and  Evaluation  Plan.  Often  these 
MOPs  are  closely  tied  to  the  functional  and 
performance  requirements  used  during 
developmental  testing,  but  modified  to  the  real 
world  as  discussed  earlier. 

Mission  Effectiveness  After  the  basic  aircraft  and 
support  functions  have  been  evaluated,  OT&E 
begins  to  explore  whether  the  total  weapon  system 
can  be  effective  to  perform  its  mission.  In  many 
instances  this  is  the  phase  when  the  understanding 
of  that  mission  begins  to  evolve  into  operational 
plans.  This  phase  includes  final  Human  Factors 
evaluations,  reliability,  and  measurements  of 
operating  limits  such  as  mission  duration  and 
climate  extremes.  In  the  US  model,  this  phase 
typically  completes  the  MOPs  while  it  begins 
evaluation  of  the  Measures  of  Effectiveness, 
(MOEs). 

Integrated  Sorties  The  final  phase  of  OT&E 
provides  evaluation  of  complete  mission  scenarios. 
In  this  phase  all  components  of  the  system  are 
brought  together  and  the  system  is  integrated  into 
the  final  force  structure.  The  scenarios  are 
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constructed  to  evaluate  the  complete  cycle 
beginning  with  crew  training  systems,  mission 
planning,  staging,  coordination  with  other  units, 
interoperability,  mission  performance,  post-flight 
activities  such  as  maintenance  and  turn-around, 
and  suitability  issues.  This  is  generally  the  phase 
where  the  end-users  begin  trying  limited 
deployments  with  the  aircraft,  and  experimenting 
with  alternate  employment  of  the  system.  The  final 
evaluation  of  the  MOEs  is  generated  during  this 
phase. 

Feedback  The  Operational  Test  and  Evaluation 
Test  Report  provides  the  basis  for  long-term  use 
and  maintenance  of  the  system.  Typically  a 
contract  is  in  place  for  Contractor  Logistic  Support 
(CLS)  which  will  provide  for  Correction  of 
Deficiencies  (COD)  found  during  OT&E. 
Additionally,  planning  begins  for  future 
improvements  based  on  ideas  and  uses  developed 
during  OT&E.  Developers  and  users  also  use  this 
phase  to  collect  Lessons  Learned,  both  as  they 
apply  to  the  development  and  testing  and  to  the 
system  itself.  Maintenance  procedures,  tech  pubs, 
users  manuals,  and  checklists  are  updated  with 
information  developed  during  OT&E.  And  finally  the 
ConOps  and  mission  of  the  system  are  refined, 
ultimately  completing  the  cycle. 


Feedback 


ConOps 


^OPs 
System  Assessment 


MOEs 


Mission  Effectiveness 
Integrated  Sorties  ' 


Figure  2  -  US  Operation  Test  Phases 


The  UK  Definition  -  Operational  Trials 

Operational  Trials  (OT)  are  designed  to  give  the 
Military  Service  units  hands-on  evaluation  of  the 
aircraft/weapon  system.  OT  starts  with  the  Concept 
of  Operations.  The  UK  use  of  the  ConOps 
approach  during  Operational  Trials  is  almost 
identical  to  the  US  usage.  Each  new  system  is 
evaluated  for  its  suitability  and  utility  for  the  planned 
mission  environment. 

The  UK  approach  allows  the  MoD  Procurement 
Executive  (PE)  some  flexibility  in  assembling  the 
Operational  Trials  team.  The  team  can  be 
comprised  of  Contractors,  Service  Units,  DERA, 
and  other  groups  as  appropriate.  All  team 
members  may  contribute  to  planning  and  executing 
the  Operational  Trials  program.  This  leads  to  a 


more  cohesive  test  organization  with  a  better- 
defined  understanding  of  both  the  strengths  and 
weaknesses  of  the  system. 

Timing  of  Operational  Trials  may  also  vary 
depending  on  the  program.  Service  units  may  begin 
evaluating  the  system  well  before  Military  Aircraft 
release  and  may  in  fact  perform  initial  Operational 
Trials  in  parallel,  or  in  conjunction  with, 
developmental  trials.  Parallel  phasing  tends  to 
ultimately  field  a  more  mature  system  concept  at 
completion  by  finding  operational  flaws  earlier. 

UK  Operational  Trials  Progression 

System  Assessment  Operational  Trials  begin 
with  Single  Activity  Tests  (SATs)  which  concentrate 
on  performance  of  individual  systems.  These  are 
typically  functional  assessments  which  verify,  under 
slightly  different  conditions,  the  results  of  the 
development  trials. 

Mission  Effectiveness  The  next  phase  begins  to 
integrate  the  basic  aircraft  with  its  supporting 
equipment  and  personnel.  The  aircrews,  mission 
planners  and  maintainers  begin  to  evaluate 
suitability  and  military  utility.  Formal  trials  begin  to 
transition  from  SATs  into  short,  limited  Multiple 
Activity  Tests  (MATs). 

Integrated  Sorties  To  complete  OT  the  length  and 
the  scope  of  the  MATs  are  increased  until  they 
embody  complete,  mission  representative  sorties. 
This  may  include  deployment  to  a  forward 
operating  base  and  transmission  of  mission  data  to 
a  surveillance  organization.  Each  user  in  the 
mission  planning,  execution,  and  maintaining 
process  has  an  opportunity  to  evaluate  the 
effectiveness  of  the  system  for  the  planned 
operational  employment. 

Feedback  The  Operational  Trials  structure  allows 
for  continuous  feedback  to  the  contractor  and  the 
service  units  about  the  progress  of  the  testing  and 
the  system.  Correction  of  Deficiencies  may  be 
performed  throughout  OT.  When  the  MoDPE 
deems  that  the  system  is  sufficiently  mature  and 
meets  an  acceptable  level  of  requirements,  the 
system  is  released  to  the  service  units. 


Feedback 


Integrated  Sorties' 


Figure  3 -UK  Operation  Test  Phases 
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Concepts  Not  Words 

The  above  definitions  indicate  clearly  that  the  major 
differences  between  the  US  and  UK,  at  least  at  the 
top  level,  are  the  titles  of  the  tests  and  test  criteria. 
There  is  a  fundamental  progression  of  testing  which 
can  be  used  to  map  the  US  MOPs  and  MOEs  to 
the  UK  SATs  and  MATs.  The  underlying  concepts 
are  the  same  and  laid  out  in  the  same  order: 

•  System  Assessment 

•  Mission  Effectiveness 

•  Integrated  Sorties 

•  Feedback 

Why  then  is  it  so  difficult  to  bring  these  approaches 
together  in  practice?  National  and  company 
arrogance  is  certainly  part  of  the  problem.  There  is 
a  decided  US-centric  and  Anglo-centric  attitude 
apparent  when  dealing  with  either  country  which 
brings  a  level  of  confrontation  into  the  day-to-day 
dealings  on  joint  programs.  Also,  within  the 
contractor  community  there  is  an  attitude  which 
routinely  ignores  customer  requests  and  real-world 
military  experiences.  This  is  the  “we’ll  tell  you  what 
you  need”  attitude.  None  of  these  attitude  issues 
can  be  corrected  or  even  addressed  without 
creating  an  environment  of  understanding  between 
all  of  the  parties  involved.  There  must  be  a 
common  starting  point  for  discussion  and  planning. 

Who  Does  What?  When? 


OT  crews  are  finally  introduced  to  the  system  and 
often  withdraw  most  of  their  original  objections  and 
complaints.  They  begin  to  see  the  system  as  a 
whole  with  all  the  complexity  that  the  initial  training 
couldn’t  give  them.  Unfortunately  by  this  time  bad 
press  and  political  wrangling  have  started  and  the 
aircraft  never  achieves  its  full  potential. 

As  mentioned  earlier,  in  the  UK  the  timing  and  roles 
of  operational  testers  are  much  more  flexible, 
frequently  allowing  for  some  combined  DT/OT  or  at 
least  contractor  involvement  in  testing.  This 
process  creates  a  feedback  loop  much  earlier  in 
system  development.  It  also  tends  to  lead  to  better 
training  materials,  more  hands-on  time,  and  more 
experienced  crews.  Contractors  also  develop  better 
understanding  of  the  operational  fielding 
requirements  because  they  are  directly  involved 
with  the  service  units.  With  both  contractor  and 
service  unit  involvement,  the  ConOps  is  refined  and 
updated,  ultimately  providing  a  more  effective 
weapon  system. 


US 


Contractor  &  DT  Squadron  OT  Squadron 


Op  Sauadron 

Fielded' 
System 


UK 

Contractor  &  Test  Sauadron  Servic§JJnits 

Fielded 
System 


One  starting  point  should  be  to  clarify  and 
standardize  the  roles  of  each  organization  involved 
in  operational  testing.  Some  compromise  between 
the  various  approaches  and  levels  of  participation 
needs  to  be  developed.  In  the  era  of  limited 
defense  budgets  and  downsizing,  performing 
testing  two  and  three  times  is  a  waste  of  valuable 
national  resources. 

As  discussed  earlier,  the  US  methods  typically 
require  that  the  OT  organization  be  completely 
isolated  from  the  development  effort.  When  the 
development  flight  test  phase  is  nearing 
completion,  the  operational  test  crews,  both 
maintainors  and  flight  crews,  are  sent  through 
training  using  the  contractor  developed  training 
material.  Once  they  have  completed  the  course 
work  they  are  handed  the  aircraft  and  told  to 
evaluate  it.  With  specifications  and  ConOps  in 
hand,  they  plan  and  fly  their  missions.  In  this 
environment  with  no  hands-on  time  or  history  with 
the  aircraft  and  its  nuances,  sorties  are  typically 
ineffective,  maintenance  suffers,  and  most  new 
aircraft  receive  a  poor  initial  rating.  The  contractor 
is  called  in,  in  crisis  mode,  to  “fix”  all  of  the 
deficiencies.  During  this  crisis  recovery  process  the 


Figure  4 -US  Vs  UK  Test  Team  Participation 


During  the  B-2  program  the  US  experimented  with 
this  combined  DT/OT  approach  and  was  able  to 
field  a  more  mature  aircraft  with  better  trained 
crews.  Maintainers  and  aircrews  ultimately  destined 
for  the  operational  squadron  at  Whiteman  AFB, 
were  the  initial  OT&E  crews  at  Edwards  AFB.  They 
participated  in  DT  missions  as  well  as  planning 
their  own  sorties.  An  effort  was  made  not  to 
duplicate  DT  testing  with  OT  missions  but  rather 
expand  and  enhance  what  had  been  done  before. 
When  the  aircraft  were  delivered  to  Whiteman  AFB, 
the  tactics  and  staging  strategy  of  the  aircraft  were 
fully  developed  and  the  aircraft  achieved  Initial 
Operational  Capability  (IOC)  quickly.  This 
approach  also  provided  a  solid  understanding 
within  the  development  community  of  what 
upgrades  and  contractor  logistics  support  (CLS) 
would  be  needed  long-term.  During  the  bombing 
campaign  in  Kosovo,  this  method  paid  off,  resulting 
in  the  B-2’s  stunningly  high  sortie  and  targeting 
rate.  These  results  have  never  been  achieved  in 
military  history,  and  can  be  attributed  to  the 
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effective  development,  testing,  and  training  in  the 
B-2  program. 

This  evaluation  indicates  that  the  most  effective 
operational  test  or  trials  programs  allow  early 
participation  by  the  operational  units  in  the  flight 
testing  and  continue  to  allow  contractor 
involvement  during  formal  trials.  This  provides  a 
more  enlightened  atmosphere  for  development  and 
fielding  on  both  the  contractor  and  military  sides.  It 
also  reduces  overall  test  costs  by  testing  the 
system  in  a  continuous  feedback  environment 
which  evolves  and  develops  the  weapons  system, 
training,  tactics,  and  strategy  throughout  the  trials 
program. 

Big  Words.  Bigger  Differences 

On  the  C-130J  the  plan  is  to  perform  a  Joint 
USAF/RAF/RAAF  test  program  aptly  named 
“Operationally  Representative  Strategic  Workload 
Tests”.  The  intent  is  to  demonstrate  that  the  C-130J 
workload  concept  does  not  overstress  a  three- 
person  crew.  The  proposed  concept  uses  two  pilots 
and  a  system  specialist  who  serves  as  both  flight 
engineer  and  load  master.  Previous  C-130  models 
used  a  five-person  crew:  two  pilots,  a  navigator,  a 
flight  engineer,  and  a  loadmaster. 

The  joint  test  concept  began  well  and  all  of  the 
major  plans  were  agreed  to  by  all  parlies.  Then  the 
detailed  planning  began  and  soon  it  was 
determined  that  each  test  had  to  be  flown  twice  or 
three  times  because  each  air  force  had  to 
“independently”  evaluate  the  test  points  to  achieve 
their  OT  objectives. 

Some  of  this  re-fly  was  reduced  by  various  services 
agreeing  to  ride  along  and  witness  some  tests. 
This  is  feasible  on  a  C-130,  but  not  on  a  fighter  like 
a  Harrier.  In  that  case,  the  services  would  have  to 
agree  to  mixed  crews  or  even  acceptance  of  each 
others  evaluations  during  various  flights.  While  this 
does  not  seem  unreasonable,  national  laws  may 
make  this  a  requirement.  These  requirements  are 
intended  to  protect  the  services  from  receiving 
ineffective  weapon  systems,  but  should  allow  for 
flexibility  during  joint  programs  where  each  service 
has  essentially  the  same  mission  objective  in  mind. 

The  next  major  obstacle,  which  has  yet  to  be 
resolved,  is  the  definition  of  “operationally 
representative”.  One  of  the  three  participating  air 
forces  have  stated  that,  in  a  workload  assessment, 
the  aircrews  may  not  know  in  advance  what  “tests” 
will  happen  during  the  mission.  In  addition,  the 
route  flown  cannot  be  pre-planned  because  the 
crew  must  be  free  to  make  their  own 
divert/rerouting  decisions.  Obviously,  within  the 
highly  controlled  airspace  environment  and  the 
needs  of  flight  test,  this  re-route  issue  can  become 
nearly  insurmountable. 


A  mission  scenario  might  consist  of  take-off  with  a 
40,000-pound  payload  with  an  intention  to  do  an 
airdrop  5  hours  into  the  flight  at  a  specified  military 
facility  and  a  return  to  base.  The  airdrop  test  would 
require  chase  aircraft,  range  safety  and  security 
sweeps,  etc.  The  workload  assessment  test  would 
induce  a  simulated  engine  failure  which  would 
create  the  need  to  dump  the  load  at  the  earliest 
possible  time.  It  becomes  obvious,  due  to  the  range 
safety  and  support  requirements,  that  the  cargo 
must  be  dropped  at  the  pre-planned  drop  location. 
The  aircrew  flying  the  test  does  not  get  a  choice  in 
this,  other  than  to  bring  the  cargo  home,  and  this  is 
the  issue  which  can  not  be  resolved.  Is  the  aircrew, 
knowing  from  the  outset  exactly  when  and  where 
the  cargo  will  be  dropped,  psychologically 
challenged  enough  for  this  scenario  to  be 
operationally  representative? 

One  air  force  believes  it  is,  while  one  doesn’t,  the 
third  hasn’t  expressed  an  opinion,  and  the 
contractor  can’t  complete  the  contract  until  they 
agree.  The  challenge  is  to  reconcile  the  definitions 
of  operationally  representative  and  provide  the 
support  and  mission  scenario  that  can 
accommodate  the  requirements  of  the  participants. 
Though  only  in  the  planning  stages,  the  scenario 
described  above  must  be  resolved  before  testing 
can  continue.  This  scenario  could  be  performed  in 
the  US  western  test  ranges,  where  it  could  be 
arranged  to  have  multiple  cargo  drop  options  on  a 
single  mission.  The  crew  could  then  drop  the  cargo 
at  one  of  several  locations  during  their  simulated 
emergency.  This  approach  quickly  becomes 
expensive,  because  multiple  sets  of  support  will  be 
required  to  allow  the  test  to  be  completed  with  all  of 
these  optional  ranges. 

This  method  also  does  not  address  the  core  issue 
of  what  is  truly  operationally  representative.  Even 
with  multiple  possible  drop  points,  the  crew  is  still 
limited  in  their  choice.  In  a  real  mission,  with  a  real 
engine  failure,  an  aircrew  would  drop  as  soon  as 
possible,  perhaps  trying  to  avoid  populated  areas 
or  enemy  encampments,  but  with  regard  for  little 
else.  In  the  test  environment  they  have  to  choose 
from  a  few  options  and  know  prior  to  the  mission 
that  something  will  “happen”  during  the  test  to  force 
them  to  choose  one.  They  will  be  briefed  on 
procedures  for  each  drop  range,  communications 
procedures,  etc.  Does  this  prior  knowledge 
contaminate  the  test  results?  Or  do  the  additional 
requirements  for  communication  and  coordination 
during  the  test  emergency  add  enough  to  the 
workload  to  create  the  required  level  of 
psychological  stress? 

There  are  no  easy  answers  to  this  question.  The 
participants  need  to  be  willing  to  explore  what  is 
possible  and  feasible  and  come  to  a  common 
definition.  Flexibility  in  defining  “operationally 
representative”  will  be  the  key  to  a  compromise  that 
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will  meet  both  requirements  and  budgets.  On  the 
surface,  it  would  appear  that  substitution  of  one 
type  of  workload  for  another  would  have  to  satisfy 
the  requirement.  This  philosophy  is  certainly 
consistent  with  previous  operational  testing.  No 
peacetime  simulation  can  ever  fully  recreate  the 
psychological  stress  of  war,  or  the  workload 
involved  in  a  volatile  environment.  This  means  that 
“operationally  representative”  testing  is  performed 
to  whatever  level  is  feasible  and  cost  effective  and 
that  true  crisis  situations  are  never  really  simulated. 

A  Hypothetical  Test 

In  the  world  of  operational  testing,  each  aircraft 
must  eventually  undergo  what  is  typically  the 
hardest  and  most  demanding  of  all  operational 
trials;  the  multi-ship  forward  deployment.  This  test 
requires  involvement  from  all  parts  of  the  weapon 
system  and  its  support  organization.  It  also 
integrates  the  system  into  the  operational 
organization,  such  as  other  aircraft  and  squadrons 
that  will  support  it,  or  that  it  will  support.  This  type  of 
test  also  provides  the  backdrop  for  most 
disagreements  in  joint  development  programs 
because  each  country  has  a  different  definition  of 
what  constitutes  a  Forward  Operating  Base  (FOB) 
and  how  the  aircraft  will  be  integrated  into 
operational  tactics. 

The  challenge  is  to  define  a  test  which  brings  all  of 
the  elements  together.  It  should  be  conducted  with 
a  joint  team  letting  each  member  evaluated  their 
particular  area  of  interest.  It  should  combine  as 
many  resources  as  possible.  It  should  be  based  on 
common  understanding  of  the  mission  roles  and 
responsibilities  and  use  common  terminology  and 
definitions. 

In  this  hypothetical  example,  the  aircraft  is  a  new 
surveillance  platform  which  can  gather  radar, 
optical,  and  infrared  (IR)  imagery,  but  not  at  the 
same  time.  The  aircraft  sensors  need  to  be 
reconfigured  depending  on  the  mission 
requirement.  In  the  radar  configuration,  the  mission 
system  can  quantify  and  track  individual  ground 
units  and  moving  vehicles.  The  US  surveillance 
doctrine  places  great  emphasis  on  the  quantified 
radar  data,  allowing  computers  to  do  most  of  the 
analysis  and  tracking  of  the  enemy  units,  and 
disperses  these  data  directly  from  the  aircraft  to  the 
field  units.  The  UK  doctrine  is  designed  around  an 
intelligence  organization  of  centrally  located 
analysts  that  evaluate  primarily  the  optical  and  IR 
imagery  and  then  disperses  tactical  and  strategic 
intelligence  back  to  the  field.  The  aircraft  under  test 
has  the  equipment  on  board  to  support  both  types 
of  operations  but  not  simultaneously.  The  tactics 
require  that  a  platform  is  on-station  at  least  18 
hours  out  of  24.  Each  aircraft  has  a  useful  mission 
duration  of  12  hours  which  include  ferry  from  and  to 
the  Forward  Operating  Base  (FOB).  The  FOB  is 


assumed  to  be  in  friendly  territory  but  within  2  hours 
flight  to  the  front  lines.  All  of  the  support  equipment, 
a  deployment  kit,  including  mission  planning,  data 
and  imagery  processing,  and  spares,  must  be 
transported  to  the  FOB,  along  with  the  personnel 
required  to  provide  this  support.  The  mission 
requirements  include  a  response  time  from  initial 
crisis  notification  to  full  deployment  and  station 
keeping. 

The  challenge  of  the  joint  test  organization  is  to 
reconcile  the  differing  doctrine  of  the  two  air  forces 
and  create  one  deployment  test  that  will  satisfy  the 
requirements  of  both.  The  test  is  an  integrated 
sortie  by  the  previously  described  definition.  In  the 
US  test  methodology  it  would  be  evaluating 
primarily  MOEs.  For  the  UK  the  test  is  defined  as  a 
MAT.  A  possible  scenario  to  satisfy  all 
requirements  could  be  developed  along  the 
following  lines. 

One  sortie  will  be  used  to  demonstrate  FOB 
Deployment.  This  will  include  the  quick  response  of 
the  aircraft  deployment  to  the  FOB,  drop  off  of  extra 
crew,  aircraft  quick-turn  and  on-board/enroute 
mission  planning.  The  ground  station  will  be 
transported  to  the  FOB  and  the  set-up/response 
time,  establishment  of  Main  Operating  Base  (MOB) 
command/control  links  to  the  FOB,  and  MOB/FOB 
support  for  the  deployment  will  be  evaluated. 

After  extra  crew  drop-off,  the  aircraft  will  proceed  to 
a  crisis-intervention  data  collection  mission.  The 
ConOps  defines  the  optimum  crisis-intervention 
configuration  as  radar  but  the  aircraft  is  configured 
for  optical  and  IR  imagery.  No  preplanned  mission 
will  be  available  and  the  scenario  assumes  that  the 
aircraft  is  not  in  the  preferred  configuration.  This 
mission  will  exercise  command  links  from  the  FOB 
to  the  MOB,  both  ground  station  and  aircraft 
responsiveness  to  a  highly  dynamic 
reconnaissance  environment.  Primary  objectives 
will  be  to  evaluate  crew  workloads,  Optics  package 
suitability  to  the  real-time  environment  and 
interoperability  with  various  command  and  control 
segments. 

A  second  aircraft  will  also  deploy  in  the  radar 
configuration,  several  hours  later.  This  aircraft  will 
rendezvous  with  the  first  aircraft  on-station  and 
relieve  it  with  a  voice  and  data  handover.  The  two 
aircraft  will  exchange  the  Area  of  Responsibility 
(AOR)  between  an  Optics  equipped  aircraft  and  a 
Radar  equipped  aircraft  and  will  include  all  ground 
station  and  command  coordination.  An  air-to-air 
data  transfer  is  planned  as  is  an  air-ground-air  data 
transfer.  The  primary  objectives  include  datalink 
evaluations,  crew  procedures,  checklist  verification, 
and  tactics  development. 

The  Radar  surveillance  mission  will  generate  target 
tracks  and  Synthetic  Aperture  Radar  (SAR)  data 
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which  will  be  analyzed  on  board  the  aircraft.  These 
data  will  be  used  by  the  on-board  mission 
commander  to  give  voice  only  target  co-ordinates 
for  air  and  ground  strike  support.  The  primary 
objectives  of  this  mission  will  be  to  demonstrate  the 
mission  suitability  of  the  radar  functions  for  strike 
support  and  targeting.  No  preplanned  mission  or 
Ground  Stations  will  be  used. 

On  return  of  the  Optics  aircraft  from  the  imagery 
collection,  the  Ground  Station  should  be 
operational,  all  mission/aircraft  support  equipment 
should  be  in  place,  and  a  relief  aircrew  should  be 
ready  to  take  the  aircraft.  The  aircraft  will  be 
prepared  for  a  sensor  change  to  reconfigure  it  to 
the  radar  configuration.  The  actual  sensor  change- 
out  will  be  performed  at  the  FOB.  All  support 
structures,  mission  planning,  communications  and 
data  co-ordination,  and  ground  station 
reconfiguration  will  be  evaluated.  Additional 
objectives  will  be  verification  of  maintenance 
procedures  and  crew  co-ordination.  The  sensor 
change-out  and  reconfiguration  to  radar  support 
should  be  completed  in  time  to  support  another 
sortie  to  relieve  the  original  radar  aircraft  on  station. 


Figure  5  -Joint  Deployment  Test 


This  scenario  is  highly  complex,  involves  numerous 
support  elements  and  intricate  test  planning. 
Nevertheless,  the  complexity  allows  the  test  to 
satisfy  the  mission  requirements  of  both  services 
with  one  series  of  sorties  and  a  single  deployment. 
Each  service  involved  flies  the  sorties  which  are 
most  applicable  to  their  particular  doctrine:  UK  on 
the  Optics  mission,  and  US  on  the  Radar  mission, 
with  both  services  participating  in  the  deployment, 
maintenance,  mission  planning,  and  data 
processing  operations.  By  planning  the  division  of 
labor  in  this  way,  all  of  the  overall  objectives  are 
met  and  each  service  can  evaluate  their  particular 
mission  requirements.  The  joint  operation  clearly 
emulates  a  real  crisis  situation  in  which  both 
countries  might  respond.  In  a  joint  crisis  response 
they  would  have  to  coordinate  their  data  gathering, 
tactics,  and  logistics  in  just  such  a  way.  Finally, 
using  this  approach,  both  MOEs  and  MAT  criteria 
are  satisfied  when  the  test  is  successfully 
completed. 


Recommendations 

On  future  joint  development  programs,  and  current 
programs  where  the  requirements  and  contracts 
are  flexible,  the  operational  test  and  trials  should  be 
designed  to  satisfy  the  most  requirements  and 
deliver  the  most  mature  system  possible.  This  can 
be  achieved  by  using  common  definitions  and 
organizational  structured  based  on  the  best 
practices  of  both  the  US  and  UK. 

Organization  Definition  Create  a  joint  test/trials 
organization  that  includes  the  contractor  as  well  as 
both  services.  This  provides  a  closed-loop 
feedback  system.  The  contractor  products, 
including  training  materials  and  aircraft  and 
maintenance  manuals  will  be  better  suited  to  the 
warfighters  and  maintainers  using  this  approach. 
Clearly  define  participants’  role  in  the  joint  test 
team,  and  assign  them  tasks  that  use  their 
strengths. 

Test  Timing  Begin  Operational  Test  and 
Evaluation  during  later  stages  of  development 
testing  so  major  deficiencies  are  addressed  earlier 
in  the  process.  Build  on  the  developmental  testing 
rather  than  repeating  it. 

Common  Phase  Definitions  Define  the 
operational  trials  phases  as  System  Assessment, 
Mission  Effectiveness,  integrated  Sorties,  and 
Feedback,  if  required  by  the  contract,  a  matrix  can 
be  created  which  maps  the  MOPs,  MOEs,  SATs, 
and  MATS  into  each  test  defined  in  these  phases. 

Use  Ail  Of  The  Resources  Plan  and  perform  tests 
which  take  advantage  of  the  differences  in  doctrine 
and  approach.  Not  only  does  this  allow  each 
country  to  evaluate  their  items  of  interest,  but  it  will 
also  highlight  how  the  doctrine  and  tactics 
differences  will  dovetail,  or  perhaps  conflict,  during 
real-world  joint  operations. 

Create  a  Dictionary  During  the  initial  planning 
stage,  all  team  members  should  jointly  create  and 
agree  on  common  definitions.  Terms  such  as 
“operationally  representative",  ‘^witnessing  versus 
participation”,  and  “strategic  workload”  should  be  as 
well  defined  as  the  performance  numbers  in  a 
specification.  This  prevents  disagreements  during 
planning  and  testing.  All  participants  will  be  working 
from  a  common  language. 
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Abstract 

The  United  States  Air  Force  is  changing  the  way 
it  both  acquires  and  employs  new  systems; 
precipitant  is  a  re-engineering  of  its  Operational 
Test  and  Evaluation  (OT&E)  methodology.  Two 
circumstantial  alterations  compelled  this 
methodological  adjustment.  Streamline 
acquisition  initiatives,  such  as  Advanced 
Concept  Technology  Demonstrations,  Battlelab 
Initiatives,  Evolutionary  Acquisition,  and 
Commercial  and  Non  Developmental  Items 
(CaNDI),  are  rapidly  reducing  the  timeline 
associated  with  the  development  and 
subsequent  fielding  of  new  capabilities  to  the 
warfighter.  Secondly,  the  arena  in  which  new 
capabilities  are  fielded  has  become  increasingly 
joint  and  multi-national,  as  evidenced  by  military 
actions  during  the  last  ten  years.  This  arena 
requires  an  intricate  understanding  of  how  a 
designated  weapon  system  will  be  used  and 
maintained,  as  well  as  identifying  which 
missions  this  system  will  supplement  in  combat. 

Shorter  development  times,  reduced  resources 
more  sophisticated  weapon  systems,  and 
employment  in  an  increasingly  complex 
environment  presents  operational  testers  with  a 
tremendous  challenge-cost-effectively  replicate 
the  warfighting  environment  to  simultaneously 
support  the  acquisition  decision  for  long-term 
procurement  and  provide  operationally  useful 
information  to  the  warfighter. 

As  the  US  Air  Force  organization  responsible  for 
OT&E,  the  Air  Force  Operational  Test  and 
Evaluation  Center  (AFOTEC),  has  incorporated 
a  new  approach  to  define  and  optimize 
operational  test  and  evaluation.  The  first  step, 
researches  battlefield  operations  and  the 
system-under-test  (SUT)  in  terms  of  its  design 
and  the  mission(s)  it  is  intended  to  supplement, 
and  identifies  the  interaction  with  other  systems 
required  to  successfully  accomplish  operational 


tasks  in  support  of  the  mission.  This  analysis 
identifies  the  significant  impacts  of  the  SUT  on 
the  mission,  and  the  complimentary  mission 
effects  on  the  SUT.  Given  this  foundation  of 
understanding,  AFOTEC  implements  the  second 
phase  of  the  approach.  In  this  phase  a  team 
evaluates  the  analysis  and  determines  the 
appropriate  level  of  testing  required  for  the  SUT, 
based  on  the  value  of  the  information,  impact  to 
the  mission,  test  venue  opportunities  and  of 
course,  constrained  resources. 

The  result  of  this  approach  is  an  OT&E  designed 
to  provide  both  sufficient  information  to  the 
acquisition  authority  to  support  a  full-rate 
production  decision  and  provide  insight  to  the 
warfighter  on  advantages  and  disadvantages  to 
successful  mission  accomplishment.  The  latter, 
called  Operational  Impact  Assessment  (OIA), 
may  or  may  not  be  due  to  the  introduction  of  the 
new  system  and  its  capabilities.  This  paper  will 
provide  an  example  of  this  process  and  highlight 
the  advantages,  through  increased  information, 
to  both  the  acquisition  decision-maker  and  the 
ultimate  customer,  the  warfighter. 


Background 

Operational  Test  and  Evaluation  is  an  integral 
part  of  the  acquisition  process.  OT&E  provides 
both  the  acquisition  decision  authority.  Major 
Command  (MAJCOM)  and  the  warfighter  with 
valuable  information  on  the  deployment, 
employment  and  sustainment  capabilities  of  new 
systems.  United  States  public  law  mandates 
independent  evaluation  by  operational  testers 
for  all  major  acquisition  programs.  An 
independent  report  is  published  prior  to 
production  and  fielding  of  these  defense 
systems. 

The  Business  of  OT&E 
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Figure  1:  OIA  -  The  Concept 


The  business  of  OT&E  is  to  provide  information 
to  both  the  acquisition  authority  and  the 
warfighter.  To  effectively  provide  this  relevant 
information  the  "O"  in  OT&E  must  be  the  focus. 
Any  OT&E  with  a  strong  foundation  in 
operations  will  lead  to  an  evaluation  focused  on 
the  warfighter's  needs.  Finally,  any  evaluation 
must  be  founded  in  accurate  and  complete  test 
data.  We  must  always  keep  in  mind,  however, 
that  tests  generate  data  and  the  evaluation 
produces  information. 

Operational  Impact  Assessment  (OIA) 

The  Concept 

As  seen  in  figure  1 ,  the  foundation  of  OT&E  is  to 
address  system  effectiveness  and  suitability  in 
an  operational  environment,  and  support  the 
system  acquisition  process.  The  new  approach, 
however,  is  to  expand  this  circle  to  increase  the 
value  of  OT&E  information  to  battlefield 
operations  for  the  warfighter.  Figure  2  shows 
where  OlAs  fit  in  the  overall  structure  of  test  and 
evaluation  (T&E). 

Developmental  and  Contractor  Test  and 
Evaluation  (DT&E/CT&E)  is  performed  by  the 
development  community  (contractor  and/or  the 
government).  This  aspect  of  T&E  focuses  on 
evaluating  system  performance  against 
specifications  in  the  contract  that  are  based  on 
the  user's  operational  requirement  documents 


(ORD).  OT&E,  by  law,  refers  to  (1)  the  field  test, 
under  realistic  combat  conditions,  of  any  item  of 
(or  key  component  of)  weapons,  equipment,  or 
munitions  for  the  purpose  of  determining  the 
effectiveness  and  suitability  of  the  weapons, 
equipment,  or  munitions  for  use  in  combat  by 
typical  military  users;  and  (2)  the  evaluation  of 
the  results  of  such  test.”  Hence,  traditional 
OT&E  emphasized  system-level  effectiveness 
and  suitability.  The  SUT  was  evaluated  in  an 
operationally  realistic  environment  with  a  focus 
of  determining  how  well  the  system  actually 
worked  when  operated  and  maintained  by  real 
"operators".  Considerations  outside  of  the 
stated  system  operational  requirements  were 
often  not  considered. 


Ensure  that  we  think  through  all 
aspects  of  employment  -not 
just 'does  the  system  - 
work'..:Gen  Oliver  ’ 


As  presented  in  a  January  1998  International 
Test  and  Evaluation  Association  (ITEA)  article, 
Maj.  Gen.  Cliver  (AFOTEC  commander  at  the 
time),  described  the  focus  of  OT&E  as  too 
limited  since  it  did  not  provide  adequate 
information  across  the  full  spectrum  of  the 
system  acquisition  and  employment  processes. 
The  new  focus  on  OIA  was  a  significant  change 
from  the  way  AFOTEC  had  done  business  for 
nearly  25  years. 
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Figure  2:  T&E  Focus  Areas 


OIA,  the  third  evaluation  area  in  figure  2,  is  the 
new  emphasis  at  AFOTEC.  It  is  not  limited  to  an 
evaluation  of  the  SUT.  It  includes  the  potential 
impacts  on  battlefield  operations, 
deployment/employment/sustainment 
considerations,  and  the  interoperability  issues 
introduced  by  the  complexities  of  joint  system- 
of-system  environments.  That  is,  OlAs  look  at 
other  systems  “feeding”  the  SUT  (such  as  C4I 
and  intelligence  systems),  and  other  systems 
dependent  on  output  from  the  SUT.  OlAs  can 
identify  holes  and  shortcomings  of  the  larger 
system-of-systems  for  the  warfighters.  These 
assessments  involve  an  interdependent, 


system-of-systems  viewpoint  of  joint  and/or 
multi-national  interoperability  issues  and 
operational  plan/concept  plan 

(OPLAN/CONPLAN)  observations,  in  addition  to 
traditional  OT&E.  In  short,  an  OIA  provides  the 
warfighters  (e.g.,  Commanders-in-Chief,  Joint 
Forces  Air  Component  Commanders, 
Commander  Air  Forces)  an  assessment  of  the 
system’s  contribution  to  mission  effectiveness 
and  suitability  in  a  joint,  coalition,  system-of- 
systems  operational  environment. 

In  order  to  develop  a  thorough  evaluation 
structure,  operational  testers  must  understand 
battlefield  operations  as  well  as  the  system 
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Figure  3:  Developing  the  Evaluation  Structure 


21.3 


INTEL 

1 

COM 

1 

THEATE 

C2 

1 

TRAININ 

Figure  4:  Understanding  Battlefield  Operations 


requirements  and  functions  in  an  operational 
environment  as  shown  in  figure  3. 

In  the  earliest  stages  of  AFOTEC's  involvement, 
core  teams  are  formed  to  research  and  collect 
system  information  from  various  sources  to 
include  the  mission  need  statement,  ORD, 
system  concepts  of  operation  (CONORS)  and 
maintenance,  and  system  program  office 
documentation/  discussions.  This  is  a  precursor 
to  understanding  where  the  SUT  fits  into 
battlefield  operations  and  to  understand  system 
operating  capabilities  and  limitations.  The 
operational  significance  of  the  ORD  parameters 
needs  to  be  sufficiently  understood  in  order  to 
scope  the  evaluation  to  meet  acquisition 
requirements. 

The  first  step  in  understanding  battlefield 
operations  is  to  identify  the  operational  task(s), 
the  system  under  development  will  support.  The 
next  step  is  to  breakdown  the  operational  task 
into  its  execution  and  support  elements.  An 
example  of  these  first  two  steps  is  shown  in 
figure  4. 

The  goals  of  understanding  battlefield 
operations  are  to:  (1)  identify  operational 
objectives  and  associated  operational  tasks 
supported  by  the  system,  (2)  understand  any 
significant  operational  issues  from  the 
warfighter's  perspective,  associated  with 
accomplishing  the  operational  tasks,  and  (3) 
understand  how  execution  and  support  elements 


work  together  to  support  the  accomplishment  of 
the  operational  task.  This  understanding  is 
essential  for  determining  the  following 
operational  impact  considerations: 

•  Execution  and  support  element  impacts  on 
the  operational  task 

•  Execution  and  support  elements  impacts  on 
each  other 

•  System  impacts  on  the  execution  and 
support  elements 

•  Execution  and  support  element  impacts  on 
the  system 

This  step,  called  a  Risk/Impact  analysis, 
identifies  the  major  impacts  on  accomplishing 
the  operational  task. 

Implementation  Process 

In  addition  to  developing  an  innovative  OIA 
approach,  AFOTEC  has  also  developed  a 
thorough  implementation  process  (figure  5).  A 
core  team  of  operational  testers  is  formed  to 
work  each  step  of  the  process. 

The  first  steps  in  this  implementation  process 
are  to  understand  battlefield  operations  and 
perform  a  system  introduction.  To  understand 
battlefield  operations,  the  operational  testers 
must  fully  address  joint  warfighting  issues  and 
the  operational  objectives/tasks  that  the  SUT  will 
support.  In  addition,  functional  mission 
decomposition  must  be  completed  to  fully 
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understand  battlefield  operations.  The  system 
introduction  consists  of  understanding  the 
operating  capabilities  and  limitations  of  the  SUT, 
the  significance  of  the  operational  requirements, 
and  how  the  system  relates  to  battlefield 
operations.  The  risk/impact  analysis 
performed  by  AFOTEC  identifies  the  major 
risks/impacts  associated  with  accomplishing  the 
operational  task(s). 

Once  the  major  risks  to  accomplishing  the 
operational  task  have  been  identified  a  "cause 
and  effect"  structure  is  developed  around  these 
risk  areas  to  understand  the  sufficiency  of 
conditions  that  could  lead  to  a  failure  to 
accomplish  an  operational  task.  This  process, 
called  OT&E  Focus,  identifies  the  evaluation 
objectives  since  they  normally  show  up  as 
critical  nodes  in  the  structure.  The  ORD 
parameters  and  Critical  Operational  Issues 
(COI)  are  then  mapped  into  the  cause  and  effect 
structure  since  they  must  be  addressed  to 
support  the  acquisition  process  and  to  ensure 
that  any  "system  critical"  area  has  not  been 
overlooked.  Finally,  boundaries  are  drawn 
around  parts  of  the  evaluation  structure  as  the 
initial  scope  is  proposed  based  on  the  potential 


value  of  the  information.  This  last  step  requires 
significant  test  and  operational  experience. 

During  the  options  and  cost  phase,  AFOTEC 
examines  the  major  test  options  available  (i.e., 
open-air  testing,  modeling  and  simulation,  and 
test  facilities  or  a  combination  in  a  distributed 
environment)  and  their  associated  costs.  Once 
the  test  options  are  identified,  AFOTEC 
conducts  a  cost/benefit  tradespace  analysis  to 
select  the  optimum  option  available.  Having 
determined  the  OT&E  focus,  AFOTEC  allocates 
the  appropriate  test  resources  and  transfers  the 
test  program  to  the  appropriate  field  detachment 
for  detailed  planning  and  test  execution. 

For  the  execution  phase,  AFOTEC  is 
augmenting  traditional  operational  testing 
methods  by  increasing  involvement  in  joint 
exercises  and  experiments  in  order  to  test  in  a 
more  stressing,  realistic  joint  operational 
environment.  AFOTEC  is  also  investigating 
opportunities  for  increased  integrated  testing  of 
systems  with  Army,  Navy,  and  Marine 
Operational  Test  Agency  test  forces.  Again,  the 
central  focus  of  operational  test  and  evaluation 
is  on  the  system’s  contribution  to  mission 
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Figure  6:  MARION  Tool  -  Mission  Decomposition 


effectiveness  and  suitability  in  a  joint,  coalition, 
system-of-systems  operational  environment. 

Implementation  Tool 

To  support  this  process,  AFOTEC  developed  a 
software  tool  called  Mission  Analysis  and  Risk 
Impact  on  Operations  Net  (MARION).  The 
MARION  tool  is  based  on  mission 
decomposition  and  considers  the  overall  mission 
that  the  SUT  supports.  Once  operational  testers 
define  test  measures,  the  MARION  tool  supports 
selection  of  the  optimum  methods  to  gather  the 
necessary  and  sufficient  test  data  to  support  the 
evaluation. 

As  seen  in  figure  6,  the  tabs  along  the  top  follow 
the  major  steps  of  the  implementation  process, 
with  understanding  battlefield  operations  and 
system  description  incorporated  into  the 
Dendritic  tab.  This  provides  a  “digital” 
representation  of  the  mission  functional 
decomposition,  allowing  the  development  of 
mission  areas  to  be  shared  between  programs. 


Also  at  this  time,  questions  in  terms  of 
overarching  mission  perspectives  and  specific 
system  requirements  can  be  ‘attached’  to  the 
different  boxes  of  the  decomposition  for  use 
later  in  developing  test  strategies. 

Recall  the  purpose  for  the  decomposition  is  to 
develop  a  common  understanding  between  all 
players.  MARION  will  augment  the  process  by 
allowing  previous  work,  for  similar  missions,  to 
be  “copied  and  pasted”  into  new  programs, 
automatically  linking  the  programs.  The  OIA 
process  can  be  applied  to  any  OT&E  program. 
The  example  shown  in  figure  6,  is  a  tracking 
system  for  patients  requiring  medical  treatment. 

The  “radio  buttons”  on  the  bottom  facilitate  the 
Risk/Impact  analysis.  Again,  this  provides  a 
methodology  for  understanding  the  importance 
between  elements  of  the  decomposition,  both  in 
terms  of  the  operational  mission  and  the  system- 
under-test.  By  viewing  the  interactions  from  the 
four  perspectives,  Element-on-Mission,  Element- 
on-System,  System-on-Element,  and  Element- 
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Figure  7:  MARION  Tool  -  Cause  and  Effect  Analysis 


on-Element,  a  broader  perspective  can  be 
generated  to  identify  the  most  important 
parameters,  in  terms  of  operational  significance, 
to  evaluate  during  OT&E. 

For  example,  as  shown  in  figure  6,  for  the 
mission  decomposition  and  looking  at  Element- 
on-System  impacts,  those  outlined  in  red 
(Transport  Patient,  Execution  C2,  and  Global 
Theater  C2)  are  rated  operationally  significant, 
and  those  outlined  in  yellow  (Medical  Treatment 
and  Comm)  are  rated  operationally  relevant. 
The  core  team  determines  these  ratings.  One 
can  relate  these  ratings  to  the  relevant  weight 
associated  with  that  element  to  overall  mission 
completion. 

Additionally,  with  Element-to-Element  impacts, 
the  tool  can  track  the  rating  of  one  element  on 
another.  The  rationale  for  this  rating,  and  any 
associated  questions  and  measures  are  stored 
in  a  database  for  use  in  the  Cause-Effect 
Analysis  and  to  determine  the  coverage 
associated  with  specific  test  strategies. 

The  Cause-Effect  analysis,  with  a  basis  from 
Theory  of  Constraints,  links  mission  failures  to 
the  previous  steps.  The  data  from  the  previous 


steps  is  available  to  help  make  the  associations, 
and  the  tool  also  allows  comments  to  be 
associated  with  each  of  the  links.  The 
knowledge  gained  from  the  previous  steps  is 
applied  to  generate  the  diagram  in  figure  7.  In 
addition,  the  information  stored  in  the  database 
is  available  for  review,  and  can  be  modified  at 
this  stage  to  support  the  evaluation  structure 
development. 

The  final  development  of  the  MARION  tool 
requires  a  methodology  to  gather  data  to 
substantiate  the  evaluation  objeotives  and 
structure  developed  in  the  Cause-Effect 
analysis.  The  proposed  solution  will  identify  the 
evaluation  objective  and  use  the  database 
information  to  identify  the  measures  and 
questions  generated  during  the  decomposition 
and  risk/impact  analysis.  The  options  available 
to  gather  this  data  will  be  supplied  by  the 
database,  to  include  similar  options  from  other 
test  programs.  These  are  then  aggregated  to 
define  separate  test  strategies,  with  its 
associated  attributes  of  cost,  schedule,  and 
amount  of  information  provided. 

These  strategies  can  be  reviewed  by  either 
looking  at  the  coverage  of  a  given  evaluation 
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objective  or  by  examining  a  given  test  location. 
These  really  define  two  different  constraints  to 
which  one  would  like  to  optimize  in  the  future. 

Once  several  test  strategies  have  been 
developed,  they  can  be  “ranked”  subjectively 
based  on  coverage  of  evaluation  objectives, 
number  of  Critical  Operational  Issues  answered, 
or  any  other  attributes,  such  as  cost,  resources, 
or  schedule. 


CONCLUSION 


The  real-world  operational  environment  has 
changed,  and  AFOTEC  needed  to  change 
accordingly.  Joint  Vision  2010,  AF  Global 
Engagement,  and  USAF  Doctrine  publications 
have  all  pointed  out  that  the  U.S.  Department  of 
Defense  and  Air  Force  must  operate  in  a 
joint/coalition  operational  environment.  Further, 
technology  has  provided  the  warfighter  with  a 
vastly  more  complex  array  of  hardware  and 
software  which  must  operate  together  correctly 
to  achieve  full-spectrum  dominance.  In  addition, 
today’s  joint/coalition  force  employment 
packages  are  much  more  integrated  and 
interdependent  (e.g.  support  aircraft,  satellites, 
C4ISR,  etc).  The  individual  SUT  most  often 
requires  inputs  from  other  systems  and  provides 
information  to  other  weapon  systems  as  well. 
In  response  to  the  changes  in  warfighting  and 
system  complexity,  AFOTEC  developed  and 
implemented  an  initiative  to  provide  the 
warfighter  key  insights  into  mission 
accomplishment.  This  initiative  included  an 
implementation  process,  a  software  tool,  and 
modified  test  report  with  separate  sections 
devoted  to  system  evaluation  and  OlAs.  The 
systems  evaluation  section  provides  AFOTEC’s 
traditional  assessment  of  the  system’s 
operational  effectiveness  and  suitability  for  the 
hands-on-operator/maintainer,  as  well  as  its 
capabilities  to  meet  ORD  threshold 
requirements.  The  OIA  section  provides  an 
assessment  of  potential  force 
employment/CONOPS  issues  affecting  the 
deployment,  employment,  and  sustainment  of 
forces,  weapon  systems,  and  capabilities. 

AFOTEC  is  a  key  partner  with  the  acquisition 
and  user  communities,  and  these  changes  will 
help  ensure  the  best  weapon  systems  are 
provided  to  our  warfighters. 
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